* state: only load when needed
* state: don't use separate load/dump
* checkout: group link removal logic together
* state: use diskcache.Cache
* staging: use pbar and save to state
* state: don't pass the whole repo to __init__
* staging: explicitly pass state object
* fs: get rid of odb property
* state: don't close the connection to cache
* repo: close: close state
We try to optimize `tree.exists` calls and probably few others
in that they either look directly into the workspace or,
to the cache without running graph checks. It does not seem
to be possible just to run graph checks on `find_outs_by_path`
due to those optimizations.
So, that's why, the `collect` also does a graph check for this
reason.
Fixes#5027Fixes#4010
* state: add simple reentrance
* state: pass Repo instead of LocalCache
* state: use granularly
Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
* plots: add -e/--experiment option for plots diff
* experiments: add method to get baseline for a given experiment
* plots: default to diff against baseline instead of workspace/head for experiments
* use exact match for experiments RE since name-rev may return relative
commits
* trees: BaseTree.walk should handle errors like os.walk
* use onerror callback if exceptions should be logged or raised
* repo.ls: use onerror for tree.walk
* remote: move constants into trees
* remote: replace Remote() Cache() helper functions
* output: update for remote changes
* cache/data_cloud: update for remote changes
* dependency: update for remote changes
* stage: update for remote changes
* remote: move remaining non-index methods into tree
* tests: update remote unit tests
* remote: LocalCache needs to lock remote index, not cache index
* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH
* tests: repo tree open needs state context
* remote: make Remote.gc a class method
* operates on obj.tree, so that GC can be done with both remote and
cache instances
* bug fixes
* tests: update test_data_cloud
* bug fixes
* tests: update func tests
Currently `RepoTree.open` will throw FileNotFoundError if you have a
file that is tracked by dvc, but which has no md5 or a cache for which
is missing, even though you have the file itself in your workspace.
This corresponds very nicely to the `--no-commit` or `--no-exec`
scenario, where you might not want to commit your artifacts to cache
just yet.
Fixes#3974
* addressing: allow stage name addressing without colon
* repo: reorganize imports
* refactor/reorganize collect_granular
* add tests for stage addressing
* fix ds and cc issues
* fix unpack error in test
* tests: stage string representations
* exception: add error message
Also remove `cause` when throwing NoOutputOrStage, as
CheckoutErrorSuggestGit will start chaining all the messages
in cause of the exception.
* rename NoOutputOrStage NoOutputOrStageError
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* DvcTree: use out.dir_cache instead of out.collect_use_dir_cache
* RepoTree: support filtering out dvcfiles during walk()
* RepoTree: support recursive isdvc()
* remote: support computing checksums for tree objects
* RepoTree: pull dir contents on walk() if in `fetch` mode
* tests: add func test for DVC dir pull on RepoTree.walk()
* DvcTree: support granular open()
* RepoTree: add get_file_checksum()
* remote: revert checksum changes
* remote: add tree support for cache.save()
* tests: test saving directly to cache from RepoTree
* remote: local cache needs its own WorkingTree instance when repo.tree is
GitTree
* RepoTree: state context should be entered by open()/walk() caller
* RepoTree: only pull dir contents when cache has changed
* tree: update tree tests
* fix ds warnings, update comments
* fix windows tests
* run: add --plot/--plot-no-cache
Part of #3409
* plot: move show/diff to API
* plots: show marked plots by default
Involves some refactoring to make plots comply with the rest of the code
base.
* dvc.yaml: use top-level metrics and plots
* dvc.yaml: add support for plot template
* plots: add support for templates in dvc.yaml
* dvc.yaml: allow metrics/plots to have persist/cache
* plots: render single page by-default and introduce --show-json
* plot: use "working tree" instead of "workspace"
Just to sync this with the rest of the codebase. We should consider
renaming it to "workspace" everywhere, but for now I opt to not touch
it to not break something that depends on it.
* plots: rename -f|--file to -o|--out
Better corresponds to other commands like `import`.
* plots: use RepoTree and brancher instead of api
* stage: get rid of OutputParams
* tests: plots: add simple diff test
* plots: use `--targets` instead of `--datafile`
Makes it comply with `dvc metrics/params` CLI options.
* tests: unit: plots: add --show-json test
* stage: cleanup exception for format error
Based on feedback from the users.
* lockfile: use relpath in corruption error
* DvcTree: support stream or fetch on open()
* tests: test DvcTree dir cache support for walk()
* RepoTree: finish implementation of BaseTree functions
* repo: use RepoTree/DvcTree.open() for repo.open_by_relpath
* repo: only use RepoTree when needed for now
* fix docstring
* RepoTree: only use DvcTree.open for DVC outs
* dvc: rename plot to plots
Follows `dvc params/metrics` convention and is needed in preparation for new
commands and some refactoring.
Part of #3409
* completion: rename plot -> plots
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
* Only reproduce steps once
* Add test
* Fix linter nasties
* Got the order wrong
* Check for previous run
* Formatting
* Do the de-duplication in _reproduce_stages
Co-authored-by: Charles Baynham <[email protected]>
* repo: separate dir cache and file cache in memory
- `used_cache()`/`get_used_cache()` in repo/stage/output now return
tuples of (dir_cache, file_cache) instead of one flat/merged cache
* update tests for new get_used_cache behavior
* remote: if .dir checksum exists on remote, assume contents also exists
- affects all commands which use `cache_exists()` (remote status)
* push: only upload .dir file after full file contents has been uploaded
* gc: always remove .dir checksums first
* functional tests for push/gc
* repo: support nesting dir caches in NamedCache
* remote: NamedCache updates
* Fix tests
* Fix deepsource warnings
* tests: use pytest mocker fixture
* Update dvc/cache.py
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
* dvc: add Repo._skip_graph_checks flag
Will skip modified graph checks on add, run and import. The aim is to
avoid collecting stages and building graph on these commands.
The skip_checks flag is aimed to be private, but let people use dvc on
big repos without monkey patching or writing yaml files themselves.
* dvc: explain skip_graphs_checks flag in a comment
* test: that skip_graph_checks works
* repo: improve hack description and test
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev
Batuhan Taskaya
Saugat Pachhai
Alexander Schepanovski
Peter Rowlands (변기호)
Casper da Costa-Luis
charlesbaynhamandCharles Baynham
Paweł Redzyński
Mr. Outis