* erepo: load external repos directly from git tree
* erepo: add fetch_external
* remote/RepoTree: add callback for keeping track of download counts when
saving tree to cache
* repo: use erepo.fetch_external() in repo.fetch()
* erepo: return cache save_infos in fetch_external
* DependencyRepo: use erepo.fetch_external and cache.checkout instead of
pull_to
* erepo: remove pull_to
* DvcTree: support granularity for isdir()
* DvcTree: fix indentation bug
* RepoTree: add copytree() method
* LocalRemote: use working tree for local cache walk_files
* repo: state should be noop for GitTree based erepos
* erepo: add get_external()
* test_get: remove unneeded caplog check
* erepo: add get_rev()
* tests: fix erepo test bug
* ls: use RepoTree
* remote: support trees in _collect_dir
* remote: support trees in get_dir_checksum
* RepoTree: fix dir_cache relpaths in windows
* remote: fix duplicated fetch bug
* utils: don't skip md5 dos2unix check for git blobs
* otherwise md5 checksums will not match when comparing git version to
cached filesystem version on windows
Fixes: #3678
When file has been already tracked under git, OutputAlreadyTrackedError gets raised.
This commits add git command suggestions to help user to stop tracking file from git,
so in future commits they can use DVC for the file.
Co-authored-by: Puneetha Pai <[email protected]>
* DvcTree: use out.dir_cache instead of out.collect_use_dir_cache
* RepoTree: support filtering out dvcfiles during walk()
* RepoTree: support recursive isdvc()
* remote: support computing checksums for tree objects
* RepoTree: pull dir contents on walk() if in `fetch` mode
* tests: add func test for DVC dir pull on RepoTree.walk()
* DvcTree: support granular open()
* RepoTree: add get_file_checksum()
* remote: revert checksum changes
* remote: add tree support for cache.save()
* tests: test saving directly to cache from RepoTree
* remote: local cache needs its own WorkingTree instance when repo.tree is
GitTree
* RepoTree: state context should be entered by open()/walk() caller
* RepoTree: only pull dir contents when cache has changed
* tree: update tree tests
* fix ds warnings, update comments
* fix windows tests
* Renamed --outs-only to --dvc-only
* Changed outs_only to dvc_only
* replace output_only with dvc_only
* Restyled by black
Co-authored-by: Restyled.io <[email protected]>
* run: add --plot/--plot-no-cache
Part of #3409
* plot: move show/diff to API
* plots: show marked plots by default
Involves some refactoring to make plots comply with the rest of the code
base.
* dvc.yaml: use top-level metrics and plots
* dvc.yaml: add support for plot template
* plots: add support for templates in dvc.yaml
* dvc.yaml: allow metrics/plots to have persist/cache
* plots: render single page by-default and introduce --show-json
* plot: use "working tree" instead of "workspace"
Just to sync this with the rest of the codebase. We should consider
renaming it to "workspace" everywhere, but for now I opt to not touch
it to not break something that depends on it.
* plots: rename -f|--file to -o|--out
Better corresponds to other commands like `import`.
* plots: use RepoTree and brancher instead of api
* stage: get rid of OutputParams
* tests: plots: add simple diff test
* plots: use `--targets` instead of `--datafile`
Makes it comply with `dvc metrics/params` CLI options.
* tests: unit: plots: add --show-json test
* stage: cleanup exception for format error
Based on feedback from the users.
* lockfile: use relpath in corruption error
* dvc: rename plot to plots
Follows `dvc params/metrics` convention and is needed in preparation for new
commands and some refactoring.
Part of #3409
* completion: rename plot -> plots
* stage: extract unnecessary methods out of Stage class
* stage: change naming of functions
* stage: extract dump equal to utils
* stage: extract computation of md5 to utils
* stage: extract command run related logic in run.py
* stage: reorganize run()
* stage: boolean cosmetic changes
* stage: reorder save() and extract some into func
* stage: refactor status() into smaller functions
* extract resolve_wdir into utils
* merge _changed() and changed()
* cosmetic changes
* extract dump to utils
* extract import updates/sync to imports.py
* remove meta from PipelineStage
* fix imports, params load
* make run_stage a function, fix mocks in tests
* use original cmd
* rollback imports on decorators
* fix ds and cc
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
Note: This does not change default for `pipeline` as
it cannot show multiple DAGs at the same time. If we
are able to show multiple DAGs, we can just default to
`dvc.yaml`.
* pull/fetch/push: add stats
* fix typo: s/logging/logger
* pull: make stats detailed, with summary at the bottom
* tests: mock properly, set return_value
* add a line space
To be consistent with how git does it (e.g. objects) as well as with
future cache structure where we plan on adding `dirs`/`files`/`objects`
subdirs in the future.
* dump: lockfile is dumped deterministically
The dump is no longer deterministic/dependent on the pipeline file,
but is sorted based on file names in outs, deps or params.
Also, the params inside each files are also sorted based on name.
However, the objects inside params are not sorted deterministically
as I think it's too much to sort that, and is not easy (considering
the types of objects it might hold, eg: lists, objects, etc).
This will also provide ordered dumps for Python3.5
* fix windows 3.8 test
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Unlike old `_check_missing_deps`, this also verifies that we are able to
save more complex dependencies such as parameters, where we not only
care about the config file, but also about the parameters in it.
Fixes#3707
This patch introduces `.dvc/cache/stages` that is used to store previous
runs and their results, which could then be reused later when we stumble
upon the same command with the same deps and outs.
Format of build cache entries is single-line json, which is readable by
humans and might also be used for lock files discussed in #1871.
Related to #1871
Local part of #1234
* tests: ensure repro --downstream preserves dependency order
* repo: use reverse post-order DFS in repro --downstream
- Fixes#3602
* make test_repro and test_repro_multistage consistent
* rebase master, update multistage test
Paweł Redzyński
Peter Rowlands (변기호)
Ruslan Kuprieiev
Puneetha PaiandPuneetha Pai
Shankho Boron GhoshandRestyled.io
karajan1001andPeter Rowlands
Saugat Pachhai
Ned Letcher
nik123
Casper da Costa-Luis
Ivan ShchekleinandJorge Orpinel