* repo: separate dir cache and file cache in memory
- `used_cache()`/`get_used_cache()` in repo/stage/output now return
tuples of (dir_cache, file_cache) instead of one flat/merged cache
* update tests for new get_used_cache behavior
* remote: if .dir checksum exists on remote, assume contents also exists
- affects all commands which use `cache_exists()` (remote status)
* push: only upload .dir file after full file contents has been uploaded
* gc: always remove .dir checksums first
* functional tests for push/gc
* repo: support nesting dir caches in NamedCache
* remote: NamedCache updates
* Fix tests
* Fix deepsource warnings
* tests: use pytest mocker fixture
* Update dvc/cache.py
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
* imp: dedup and parallelize external fetches
This prevents same fetches for external artifacts, groups them by source
and fetches in parallel.
* imp: test that fetch works on non workspace deps
* dvc: fix branch suffixes in status/push/pull
* gc: add --all-commits flag
* gc: fix gc warning
Say that we always leave things used in the working tree
* dvc: dedup used cache
This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts
* dvc: delay checksum names concatenation
In this mode, dvc will set appropriate file/dir permissions to support
sharing cache between different users within the same group.
Fixes#2287
Signed-off-by: Ruslan Kuprieiev <[email protected]>
1) split Project's methods into separate files;
2) rename Project into Repo(similar to gitpython's Repo)
Signed-off-by: Ruslan Kuprieiev <[email protected]>
flake8 provides some additional features like unused vars check, etc, so
it makes total sense to use it as well.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This is an opt-in mode, that will make data in the workspace
read-only, thus protecting cache from corruption when hardlinks or
symlinks are used. User can use `dvc unprotect file` command, to
replace read-only link to cache with an editable copy. We can use
this for now to decide if we want to go with a `unified workflow`
in the future or if we want to set unprotected `reflink,copy`
by default.
Fixes#799
Signed-off-by: Ruslan Kuprieiev <[email protected]>
From now on we allow specifying standalone cache dirs as well as
specifying linking method(copy, symlink, hardlink, reflink) for it.
If 'cache type' is not specified in the config, dvc will try all of
them starting from reflink all the way down to copy.
Signed-off-by: Ruslan Kuprieiev <[email protected]>