* tests: ensure repro --downstream preserves dependency order
* repo: use reverse post-order DFS in repro --downstream
- Fixes#3602
* make test_repro and test_repro_multistage consistent
* rebase master, update multistage test
* remote: use checksums instead of paths when filling dir statuses
* remote: prefer using str paths over PathInfo for performance reasons
* only use string paths for RemoteLOCAL
* only optimize calls made from RemoteLOCAL.cache_exists
* fix ds warning
* use os.path in local
* remote: re-add checksum_to_path() to return string paths when applicable
- cloud remotes still default to using PathInfo's
* cache fspath string
* use abspath in checksum_to_path
- if path is not relpath from cwd or abspath, posix lstat() syscall
runtime doubles (from calculating relpath from cwd)
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* progress: add main push/pull bar
Fixes#3452
Related #1840
Related #3565
* adjust desc
* fix merge
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Current implementation doesn't work with directories, which results in
hard-to-debug bugs like #3561. The bug that made us walk through index
manually is now fixed in gitpython, so we can simply use `ls_files` from
now on. Added some tests to prevent this from happening in the future.
Fixes#3561
* Convert all paths in config to posix-style
* Use PathInfo for consistancy, add unit test and change functional tests
* Exclude Windows test on Linux
* Don't patch Config for all the tests!
* Don't do the linux test on Windows either
* De-hackify the unit test, make the functional test more explicit
* tests: adjust tests
Co-authored-by: Charles Baynham <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Only reproduce steps once
* Add test
* Fix linter nasties
* Got the order wrong
* Check for previous run
* Formatting
* Do the de-duplication in _reproduce_stages
Co-authored-by: Charles Baynham <[email protected]>
* remote: locally index list of checksums available on cloud remotes
* fix review issues
- use pytest fixtures
- use sha256 digest
- s/fd/fobj/
- check more specific IO errors
* remote: use .dir checksums to validate index
* command: add --drop-index option for push/pull/status
* tests: --drop-index unit tests
* remote: make index threadsafe
* remote: force re-index after gc
* only save/dump index once per command
* remote: add helper index functions update_all/replace_all
* remote: store index as sqlite3 database
* fix deepsource warnings
* rename --drop-index to --clear-index
* rename index.invalidate to index.clear
* functional tests for remote.index
* fix index not being updated on status -c
* remote: add index.intersection()
* remote: use index when making assumptions about remote .dir contents
* add missing func tests
* do not index standalone files
* cleanup index test fixture
* fix deepsource warnings
* update autocompletion scripts/cli help message for --clear-index
* tests: remove unused mocker
* push: only index successfully pushed dirs
- do not index files on partial push/upload
* remote: revert behavior to trust .dir files on remote
- skip unnecessary index check for dir contents if .dir file exists on
the remote
* fix missing RemoteIndexNoop functions
* separate index validation from status dir exists query
* bugfix: include indexed checksums for all cache_exists cases
* review fix: cleanup debug messages
* tests: fix DS warning
* remove --clear-index
* remove unused index functions
* add: -f arg name from FILE to <filename>
to match dvc.org cmd ref
* dvc: cmd arg names (metavars) to match docs
for #3626
* metrics: update help output to indicate supported types
per #3409
and for #3572
* dvc: reorder subcommands to match docs
for #3608
Now, pipeline stage files can house multiple stages, and separate lockfiles
are created which has the checksums, whereas Dvcfile will be clean and human
readable and editable. The *.dvc files will be generated for output files.
It is available via a hidden flag: -n.
Fixes#1871
PR: #3584
This is useful as it is in `dvc ls` implementation, but this is also a
prerequisite for `RepoTree` that will combine git(workspace) files and
dvc files into one tree, allowing things like `dvc metrics` and
`external_repo` to have a universal interface to access whatever
file/dir they want without cumbersome git/dvc path detection.
Using it to load dirs takes around 50 sec compared to 0.4 with simple
`replace()`. Path objects are great, but not when you need lots of them.
Related to #3635
* remote config validation
Fixes#3552
* solving config test fail
* Validation problem in config merging
sloving validation problem in config merging from different levels.
Only those config in lower levels can affect the validation process.
* reduce Cognitive Complexity
* Two request changes
1. Duplicate load current level of config
2. rewrite test to pytest style
* black it
* config: don't use all_levels
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* repo: separate dir cache and file cache in memory
- `used_cache()`/`get_used_cache()` in repo/stage/output now return
tuples of (dir_cache, file_cache) instead of one flat/merged cache
* update tests for new get_used_cache behavior
* remote: if .dir checksum exists on remote, assume contents also exists
- affects all commands which use `cache_exists()` (remote status)
* push: only upload .dir file after full file contents has been uploaded
* gc: always remove .dir checksums first
* functional tests for push/gc
* repo: support nesting dir caches in NamedCache
* remote: NamedCache updates
* Fix tests
* Fix deepsource warnings
* tests: use pytest mocker fixture
* Update dvc/cache.py
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
* refactor stage into dvcfile
* Split create_stage out of dvcfile to Stage::create() and overwrite
* Remove logging from Repo::run
* tests: stop using --remove-outs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Current logic is extremely complex and messy, hiding a lot of bugs
inside. There are pretty much only 2 scenarios for stage creation:
1) `dvc add/import/etc` when dvcfile is created beside the output file
2) `dvc run` when dvcfile is created at the place of execution
both are transparently affected by `--fname` flag. Since we've removed
`--cwd` complexity already, there is no reason to generalize stage path
resolution for those two scenarios.
Windows (or rather NTFS) doesn't allow deleting read-only files, which made us
try to chmod +w first. But that is not necessery on Linux and Mac, and
has led to unnecessary cache file mode changes, which was causing issues
in shared cache scenario when a user that doesn't own a file was trying
to automatically protect it.
Fixes#3618
Texttable will try to detect the column type automatically, which
results in a low precision of the float that it is showing. We need to
disable that auto-formatting and just use `str(val)` instead.
Peter Rowlands (변기호)
Saugat Pachhai
nik123
Ruslan Kuprieiev
Jorge Orpinel
Marcel Ribeiro Dantas
Ivan Shcheklein