Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,687 Commits 7 Branches 579 Tags
2.5.4
Commit Graph
6687 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev 67c46b98dd dvc: bump to 2.5.4 2.5.4 2021-07-08 23:02:16 +03:00
Ruslan Kuprieiev bd5809572e setup: relax requirements for importlib-metadata 2021-07-08 23:01:48 +03:00
Ruslan Kuprieiev 05832c81fc dvc: bump to 2.5.3 2.5.3 2021-07-08 19:34:26 +03:00
Ruslan KuprieievandSaugat Pachhai 86df89e3a8 pkg: don't build universal wheels (#6299) 
Kudos @skshetry for discovering that we've been publishing py2.py3
wheels on pypi and researching the cause.

Co-authored-by: Saugat Pachhai <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
 2021-07-08 19:33:52 +03:00
Ruslan Kuprieiev c33776eaee dvc: bump to 2.5.2 2.5.2 2021-07-08 18:30:11 +03:00
kareemjanoandpre-commit-ci[bot] 22781f5ef0 fs: s3: don't forget to pass verify option (#6298) 
* Update s3.py

pass verify flag when creating a session

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-07-08 21:12:36 +05:45
Ruslan Kuprieiev ae478ccdc0 dvc: bump to 2.5.1 2.5.1 2021-07-08 17:48:15 +03:00
Batuhan Taskaya 87b49fd9f9 staging: optimize the number of info() calls (#6281) 
* staging: optimize the number of info() calls

* fs: make info() give details about the object
 2021-07-07 22:53:43 +03:00
Batuhan Taskaya f90b9cdee1 doctor: show versions for fs requirements (#6273) 
* doctor: show versions for fs requirements

* plugins => supports
 2021-07-07 21:07:16 +03:00
Christian ClaussandSaugat Pachhai 40f24e5668 pre-commit: introduce codespell check 
* codespell --ignore-words-list="ba,datas,fo"

* up-to-date -> uptodate

* Add pre-commit hook...

* Pre-commit needs quotes?

* Single quotes?

* Arrrgggsss

* Update .pre-commit-config.yaml

Co-authored-by: Saugat Pachhai <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
 2021-07-07 17:14:15 +02:00
Hadas Arikandpre-commit-ci[bot] 9d9ff10be1 diff: only show states with values in summary (#6229) 
* Fixed diff summary to only show states with values.

* Removed redundant comma by using join.

* Swithced for loop to generator for better readability.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-07-06 21:03:32 +09:00
Batuhan Taskaya 98f7d1e481 objects: hash_file: retrieve size from hash_info (#6284) 
* objects: hash_file: retrieve size from hash_info

* objects: drop Tree.size implementation
 2021-07-06 10:56:13 +03:00
Saugat Pachhai 9cf22d3edc add: reduce os.fspath calls on recursive (#6287) 2021-07-06 06:43:56 +00:00
Saugat Pachhai 4b6c6a0e79 add: fix progress bar on single target (#6288) 2021-07-06 12:28:16 +05:45
Saugat Pachhai 6bccb21af6 prevent-tqdm-interleaving (#6280) 2021-07-06 13:37:37 +09:00
Saugat Pachhai beb6277d13 cleaning up add, progress bars and status (#6224) 
* cleanup code

* fix progress bar updating message even when index > total

* cleanup
 2021-07-06 01:08:35 +03:00
Kurian Benoyandpre-commit-ci[bot] 2485779d59 run: update -h message (#6197) 
* update dvc exp run -h message

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2.5.0 		2021-07-03 23:02:19 +03:00
Ruslan Kuprieiev 247b9c8be8 dvc: bump to 2.5.0 2021-07-03 22:34:11 +03:00
Saugat Pachhai 8d5f9ab6e5 add: do not delete stage files before add (#6239) 
* add: do not delete stage files before add

Because of the way we collect stages and cache them, we were not able to
collect them for the `add` without removing them from the workspace.
As doing so, we'd have two same/similar stages - one collected from the
workspace and the other just created from the `dvc add` in-memory.

This would raise errors during graph checks, so we started to delete them
and reset them (which is very recently, see #2886 and #3349).

By deleting the file before we even do any checks, we are making DVC
fragile, and results in data loss for the users with even simple mistakes.
This should make it more reliable and robust.

And, recently, we have started to keep state of a lot of things, that by
resetting them on each stage, we waste a lot of performance, especially
on gitignores. We cache the dulwich's IgnoreManager, which when resetted
too many times, will waste a lot of our time just collecting them again
next time (see #6227).

It's hard to say how much this improves, as this very much depends on
no. of gitignores in the repo (which can be assumed to be quite in number
for a dvc repo) and the amount of files that we are adding
(eg: `-R` adding a large directory).

On a directory with 10,000 files (in a datadet-registry repo),
creating stages on `dvc add -R` went from 64 files/sec to 1.1k files/sec.

* add tests

* make the test more specific
 2021-07-03 22:02:06 +03:00
Ruslan Kuprieiev 07f5f6b633 ssh: don't use path_info for connections (#6259) 2021-07-03 21:54:14 +03:00
Ruslan Kuprieiev aa255a1102 output: optimize obj collection (#6277) 
* output: str only once

Calling `__str__` takes a long time and if we are dealing with a large
dataset, it will result in a very substantial time. For example,
locally, for 700K objects, it took 20sec more.

Related to #6276

* output: don't use os.path.join

Takes around 2sec for 700K objects.

* output: use fs.sep

* output: remove unused sep
 2021-07-03 21:53:55 +03:00
Peter Rowlands (변기호) cfcb7e1294 pygit: handle annotated tags in resolve_refish (#6274) 2021-07-02 18:00:47 +03:00
Saugat Pachhai b8d9800e3c separate validation and transformation of args from the add func (#6220) 
* separate validation and transformation of args from add func

* make tests pass for Python3.6

* cleanup

* use dict for attr-access and patch boundarguments.arguments
 2021-07-01 21:15:02 +05:45
dependabot[bot] fe46d50847 build(deps): bump google-cloud-storage from 1.39.0 to 1.40.0 (#6261) 
Bumps [google-cloud-storage](https://github.com/googleapis/python-storage) from 1.39.0 to 1.40.0.
- [Release notes](https://github.com/googleapis/python-storage/releases)
- [Changelog](https://github.com/googleapis/python-storage/blob/master/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-storage/compare/v1.39.0...v1.40.0)

---
updated-dependencies:
- dependency-name: google-cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-01 21:14:20 +05:45
Batuhan Taskaya e3ff6d54a9 fsspec: loosen the prefix check to cover both None and False (#6246) 
* fsspec: loosen the prefix check to cover both None and False

* gs: disable prefix-based search
 2021-07-01 14:59:35 +03:00
Saugat Pachhai ce45ec8a28 align metrics to right and params to left in exp table (#6254) 
Made a mistake during refactoring on styling before
 2021-06-30 16:05:17 +05:45
Miguel Méndez cc5ba985b0 import: error message when imported file does not exist (#6240) 
* Fix exception handling

* Replace exceptions with PathMissingError
 2021-06-30 15:49:15 +09:00
Ruslan Kuprieiev 87cdbf8ace dvc: bump to 2.4.3 2.4.3 2021-06-29 17:27:21 +03:00
Tomasz Pietruszka eda8cd5846 git: Keep trailing slash when checking if directory is gitignored (#6232) 
* Keep trailing slash when checking if directory is gitignored

* Calling is_ignored with correct type (could be Path-like)

* Fix test to comply to annotated types

* Fix separator issues

* Relaxed typing in is_ignored

* Added test

* Test scm

* Fix test on Windows

* Expanded test_ignored_dir_unignored_subdirs

* Even more assertions
 2021-06-29 21:49:29 +09:00
pre-commit-ci[bot] 61835f6511 [pre-commit.ci] pre-commit autoupdate (#6243) 
updates:
- [github.com/pre-commit/mirrors-mypy: v0.902 → v0.910](https://github.com/pre-commit/mirrors-mypy/compare/v0.902...v0.910)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-06-29 07:24:12 +05:45
Jon Crall e72354e98e Loosen versions for fsspec and pygtrie (#6242) 2021-06-28 23:57:49 +03:00
Ruslan Kuprieiev d1b36e36ca dvc: bump to 2.4.2 2.4.2 2021-06-28 20:19:42 +03:00
Batuhan Taskaya 6ec66ab3de output: don't lower the etag field on the schema when deserializing (#6238) 2021-06-28 20:15:58 +03:00
Peter Rowlands (변기호) 7c13897674 utils: normalize case in path_isin functions (#6236) 
* utils: normalize case in path_isin

* utils: normalize case in contains_symlink_up_to

* split platform test
 2021-06-28 19:21:44 +09:00
dependabot[bot] 1db4637f2d build(deps): bump crc32c from 2.2 to 2.2.post0 (#6234) 
Bumps [crc32c](https://github.com/ICRAR/crc32c) from 2.2 to 2.2.post0.
- [Release notes](https://github.com/ICRAR/crc32c/releases)
- [Changelog](https://github.com/ICRAR/crc32c/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ICRAR/crc32c/compare/v2.2...v2.2.post0)

---
updated-dependencies:
- dependency-name: crc32c
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-28 12:17:14 +03:00
dependabot[bot] ca3757b45b build(deps): bump urllib3 from 1.26.5 to 1.26.6 (#6235) 
Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.5 to 1.26.6.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.6/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.5...1.26.6)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-28 08:43:13 +05:45
Ruslan Kuprieiev 4e792ae61c setup: update fsspec to 2021.6.1 (#6219) 
* setup: update fsspec to 2021.6.1

* fs: local: remove temp workaround
 2021-06-24 10:06:42 +05:45
dependabot[bot] 6b13f76fdb build(deps): bump mypy from 0.902 to 0.910 (#6212) 
Bumps [mypy](https://github.com/python/mypy) from 0.902 to 0.910.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.902...v0.910)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-24 10:17:52 +09:00
Ruslan Kuprieiev 53e9460c65 staging: remove redundant exists() call (#6217) 2021-06-24 00:19:06 +03:00
dependabot[bot] dafe6e991c build(deps): bump google-cloud-storage from 1.38.0 to 1.39.0 (#6211) 
Bumps [google-cloud-storage](https://github.com/googleapis/python-storage) from 1.38.0 to 1.39.0.
- [Release notes](https://github.com/googleapis/python-storage/releases)
- [Changelog](https://github.com/googleapis/python-storage/blob/master/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-storage/compare/v1.38.0...v1.39.0)

---
updated-dependencies:
- dependency-name: google-cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-23 19:07:43 +00:00
Ruslan Kuprieiev 09946847d2 dvc: bump to 2.4.1 2.4.1 2021-06-23 16:08:10 +03:00
Saugat PachhaiandRuslan Kuprieiev 6e5e62857b Bump fsspec version (#6214) 
* Bump fsspec version

* local: use fsspec 2021.06.0 with a temp hack

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2021-06-23 16:07:32 +03:00
Saugat Pachhai b7c6df0ece Create CODEOWNERS (#6213) 2021-06-23 11:56:37 +05:45
Ruslan Kuprieiev 6779d3d431 dvc: bump to 2.4.0 2.4.0 2021-06-22 20:14:46 +03:00
pre-commit-ci[bot]andRuslan Kuprieiev 5844a0cd5b [pre-commit.ci] pre-commit autoupdate (#6167) 
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/ambv/black: 21.5b2 → 21.6b0](https://github.com/ambv/black/compare/21.5b2...21.6b0)
- [github.com/timothycrosley/isort: 5.8.0 → 5.9.1](https://github.com/timothycrosley/isort/compare/5.8.0...5.9.1)
- [github.com/pre-commit/mirrors-mypy: v0.812 → v0.902](https://github.com/pre-commit/mirrors-mypy/compare/v0.812...v0.902)

* test_requirements: add types packages

* pre-commit: add additional dependencies for mypy

* pre-commit: fix formatting

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2021-06-22 13:59:11 +03:00
Saugat Pachhai ce3a3ab100 pass size to upload_fobj (#6187) 
This allows us to select between chunked and non-chunked uploading
in Webdav, and hopefully in the http itself.
 2021-06-22 13:16:15 +03:00
yehiyam d1a219df0a repo: preserve meta field on move (#6201) 
fixes #6200
 2021-06-22 13:12:20 +03:00
Saugat Pachhai 476af100de ui: migrating rest to the ui.write (#6176) 
* complete migration to ui.write

* fix small bugs
 2021-06-21 22:14:59 +03:00
Saugat Pachhai f2998630a3 env: track DVC_IGNORE_ISATTY as a env that we support (#6202) 2021-06-21 05:35:39 +00:00
Laurens Duijvesteijn 9a7b6d2a1a serialize: Wrap UnicodeDecodeErrors (#6193) 
Also expose `encoding` and `path` as attributes, so programmatic
consumers can access it easily.
 2021-06-19 11:35:50 +05:45