Unlike old `_check_missing_deps`, this also verifies that we are able to
save more complex dependencies such as parameters, where we not only
care about the config file, but also about the parameters in it.
Fixes#3707
`get_checksum()` recomputes the checksum which might not match the
pre-recorded one. `checksum` is the one that was `save()`ed during run
and it is the one that should be used in the lockfile.
This patch introduces `.dvc/cache/stages` that is used to store previous
runs and their results, which could then be reused later when we stumble
upon the same command with the same deps and outs.
Format of build cache entries is single-line json, which is readable by
humans and might also be used for lock files discussed in #1871.
Related to #1871
Local part of #1234
* Relax python-dateutil pip version constraint to include v2.8.2. #3701
[WIP]. Attempt to relax upper constraints on the version of python-dateutil, to include 2.8.1 and 2.8.2.
Original constraint was:
"python-dateutil<2.8.1,>=2.1", # Consolidates azure-blob-storage and boto3
* Update setup.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tests: ensure repro --downstream preserves dependency order
* repo: use reverse post-order DFS in repro --downstream
- Fixes#3602
* make test_repro and test_repro_multistage consistent
* rebase master, update multistage test
* remote: use checksums instead of paths when filling dir statuses
* remote: prefer using str paths over PathInfo for performance reasons
* only use string paths for RemoteLOCAL
* only optimize calls made from RemoteLOCAL.cache_exists
* fix ds warning
* use os.path in local
* remote: re-add checksum_to_path() to return string paths when applicable
- cloud remotes still default to using PathInfo's
* cache fspath string
* use abspath in checksum_to_path
- if path is not relpath from cwd or abspath, posix lstat() syscall
runtime doubles (from calculating relpath from cwd)
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* progress: add main push/pull bar
Fixes#3452
Related #1840
Related #3565
* adjust desc
* fix merge
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Current implementation doesn't work with directories, which results in
hard-to-debug bugs like #3561. The bug that made us walk through index
manually is now fixed in gitpython, so we can simply use `ls_files` from
now on. Added some tests to prevent this from happening in the future.
Fixes#3561
* Convert all paths in config to posix-style
* Use PathInfo for consistancy, add unit test and change functional tests
* Exclude Windows test on Linux
* Don't patch Config for all the tests!
* Don't do the linux test on Windows either
* De-hackify the unit test, make the functional test more explicit
* tests: adjust tests
Co-authored-by: Charles Baynham <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Only reproduce steps once
* Add test
* Fix linter nasties
* Got the order wrong
* Check for previous run
* Formatting
* Do the de-duplication in _reproduce_stages
Co-authored-by: Charles Baynham <[email protected]>
* remote: locally index list of checksums available on cloud remotes
* fix review issues
- use pytest fixtures
- use sha256 digest
- s/fd/fobj/
- check more specific IO errors
* remote: use .dir checksums to validate index
* command: add --drop-index option for push/pull/status
* tests: --drop-index unit tests
* remote: make index threadsafe
* remote: force re-index after gc
* only save/dump index once per command
* remote: add helper index functions update_all/replace_all
* remote: store index as sqlite3 database
* fix deepsource warnings
* rename --drop-index to --clear-index
* rename index.invalidate to index.clear
* functional tests for remote.index
* fix index not being updated on status -c
* remote: add index.intersection()
* remote: use index when making assumptions about remote .dir contents
* add missing func tests
* do not index standalone files
* cleanup index test fixture
* fix deepsource warnings
* update autocompletion scripts/cli help message for --clear-index
* tests: remove unused mocker
* push: only index successfully pushed dirs
- do not index files on partial push/upload
* remote: revert behavior to trust .dir files on remote
- skip unnecessary index check for dir contents if .dir file exists on
the remote
* fix missing RemoteIndexNoop functions
* separate index validation from status dir exists query
* bugfix: include indexed checksums for all cache_exists cases
* review fix: cleanup debug messages
* tests: fix DS warning
* remove --clear-index
* remove unused index functions
* add: -f arg name from FILE to <filename>
to match dvc.org cmd ref
* dvc: cmd arg names (metavars) to match docs
for #3626
* metrics: update help output to indicate supported types
per #3409
and for #3572
* dvc: reorder subcommands to match docs
for #3608
Now, pipeline stage files can house multiple stages, and separate lockfiles
are created which has the checksums, whereas Dvcfile will be clean and human
readable and editable. The *.dvc files will be generated for output files.
It is available via a hidden flag: -n.
Fixes#1871
PR: #3584
This is useful as it is in `dvc ls` implementation, but this is also a
prerequisite for `RepoTree` that will combine git(workspace) files and
dvc files into one tree, allowing things like `dvc metrics` and
`external_repo` to have a universal interface to access whatever
file/dir they want without cumbersome git/dvc path detection.
Using it to load dirs takes around 50 sec compared to 0.4 with simple
`replace()`. Path objects are great, but not when you need lots of them.
Related to #3635
* remote config validation
Fixes#3552
* solving config test fail
* Validation problem in config merging
sloving validation problem in config merging from different levels.
Only those config in lower levels can affect the validation process.
* reduce Cognitive Complexity
* Two request changes
1. Duplicate load current level of config
2. rewrite test to pytest style
* black it
* config: don't use all_levels
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* repo: separate dir cache and file cache in memory
- `used_cache()`/`get_used_cache()` in repo/stage/output now return
tuples of (dir_cache, file_cache) instead of one flat/merged cache
* update tests for new get_used_cache behavior
* remote: if .dir checksum exists on remote, assume contents also exists
- affects all commands which use `cache_exists()` (remote status)
* push: only upload .dir file after full file contents has been uploaded
* gc: always remove .dir checksums first
* functional tests for push/gc
* repo: support nesting dir caches in NamedCache
* remote: NamedCache updates
* Fix tests
* Fix deepsource warnings
* tests: use pytest mocker fixture
* Update dvc/cache.py
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
* refactor stage into dvcfile
* Split create_stage out of dvcfile to Stage::create() and overwrite
* Remove logging from Repo::run
* tests: stop using --remove-outs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Current logic is extremely complex and messy, hiding a lot of bugs
inside. There are pretty much only 2 scenarios for stage creation:
1) `dvc add/import/etc` when dvcfile is created beside the output file
2) `dvc run` when dvcfile is created at the place of execution
both are transparently affected by `--fname` flag. Since we've removed
`--cwd` complexity already, there is no reason to generalize stage path
resolution for those two scenarios.
Windows (or rather NTFS) doesn't allow deleting read-only files, which made us
try to chmod +w first. But that is not necessery on Linux and Mac, and
has led to unnecessary cache file mode changes, which was causing issues
in shared cache scenario when a user that doesn't own a file was trying
to automatically protect it.
Fixes#3618
Ruslan Kuprieiev
Peter Rowlands (변기호)
Ivan ShchekleinandJorge Orpinel
Dmitry ChichkovandRuslan Kuprieiev
Saugat Pachhai
nik123
Marcel Ribeiro Dantas