* plots: remove --select from CLI
It needs reconsideration, we can't release it as is right now. E.g. it
expects xpath to start with `$`, but that will get interpreted by your
shell, which is bad.
* completion: update plots
* DvcTree: use out.dir_cache instead of out.collect_use_dir_cache
* RepoTree: support filtering out dvcfiles during walk()
* RepoTree: support recursive isdvc()
* remote: support computing checksums for tree objects
* RepoTree: pull dir contents on walk() if in `fetch` mode
* tests: add func test for DVC dir pull on RepoTree.walk()
* DvcTree: support granular open()
* RepoTree: add get_file_checksum()
* remote: revert checksum changes
* remote: add tree support for cache.save()
* tests: test saving directly to cache from RepoTree
* remote: local cache needs its own WorkingTree instance when repo.tree is
GitTree
* RepoTree: state context should be entered by open()/walk() caller
* RepoTree: only pull dir contents when cache has changed
* tree: update tree tests
* fix ds warnings, update comments
* fix windows tests
* Renamed --outs-only to --dvc-only
* Changed outs_only to dvc_only
* replace output_only with dvc_only
* Restyled by black
Co-authored-by: Restyled.io <[email protected]>
* run: add --plot/--plot-no-cache
Part of #3409
* plot: move show/diff to API
* plots: show marked plots by default
Involves some refactoring to make plots comply with the rest of the code
base.
* dvc.yaml: use top-level metrics and plots
* dvc.yaml: add support for plot template
* plots: add support for templates in dvc.yaml
* dvc.yaml: allow metrics/plots to have persist/cache
* plots: render single page by-default and introduce --show-json
* plot: use "working tree" instead of "workspace"
Just to sync this with the rest of the codebase. We should consider
renaming it to "workspace" everywhere, but for now I opt to not touch
it to not break something that depends on it.
* plots: rename -f|--file to -o|--out
Better corresponds to other commands like `import`.
* plots: use RepoTree and brancher instead of api
* stage: get rid of OutputParams
* tests: plots: add simple diff test
* plots: use `--targets` instead of `--datafile`
Makes it comply with `dvc metrics/params` CLI options.
* tests: unit: plots: add --show-json test
* stage: cleanup exception for format error
Based on feedback from the users.
* lockfile: use relpath in corruption error
* GitTree: add stat() function
* BaseTree: make walk_files a base tree method
* WorkingTree: add stat() method (wrapper for os.stat)
* utils.fs: support non-working tree for get_mtime_and_size
* utils.fs: add copy_fobj_to_file
* utils: support using tree.open in file_md5
* DvcTree: support stream or fetch on open()
* tests: test DvcTree dir cache support for walk()
* RepoTree: finish implementation of BaseTree functions
* repo: use RepoTree/DvcTree.open() for repo.open_by_relpath
* repo: only use RepoTree when needed for now
* fix docstring
* RepoTree: only use DvcTree.open for DVC outs
* dvc: rename plot to plots
Follows `dvc params/metrics` convention and is needed in preparation for new
commands and some refactoring.
Part of #3409
* completion: rename plot -> plots
* stage: extract unnecessary methods out of Stage class
* stage: change naming of functions
* stage: extract dump equal to utils
* stage: extract computation of md5 to utils
* stage: extract command run related logic in run.py
* stage: reorganize run()
* stage: boolean cosmetic changes
* stage: reorder save() and extract some into func
* stage: refactor status() into smaller functions
* extract resolve_wdir into utils
* merge _changed() and changed()
* cosmetic changes
* extract dump to utils
* extract import updates/sync to imports.py
* remove meta from PipelineStage
* fix imports, params load
* make run_stage a function, fix mocks in tests
* use original cmd
* rollback imports on decorators
* fix ds and cc
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
Currently we only restore `--outs`, but `--outs-no-cache` are also
useful, especially when they are metrics files
(`-M|--metrics-no-cache`). So this PR makes run-cache save uncached
files to our cache and also tries to checkout them back during the
`restore` phase.
* params/metrics: diff: implement tabulated markdown output
Via `--show-md` on `diff`s.
* table: show None on missing val
* tests: use `dedent`, add tests for markdown
* diff: add --show-md
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Note: This does not change default for `pipeline` as
it cannot show multiple DAGs at the same time. If we
are able to show multiple DAGs, we can just default to
`dvc.yaml`.
* pull/fetch/push: add stats
* fix typo: s/logging/logger
* pull: make stats detailed, with summary at the bottom
* tests: mock properly, set return_value
* add a line space
This patch introduces `.dvc/cache/stages` that is used to store previous
runs and their results, which could then be reused later when we stumble
upon the same command with the same deps and outs.
Format of build cache entries is single-line json, which is readable by
humans and might also be used for lock files discussed in #1871.
Related to #1871
Local part of #1234
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Current implementation doesn't work with directories, which results in
hard-to-debug bugs like #3561. The bug that made us walk through index
manually is now fixed in gitpython, so we can simply use `ls_files` from
now on. Added some tests to prevent this from happening in the future.
Fixes#3561
* Convert all paths in config to posix-style
* Use PathInfo for consistancy, add unit test and change functional tests
* Exclude Windows test on Linux
* Don't patch Config for all the tests!
* Don't do the linux test on Windows either
* De-hackify the unit test, make the functional test more explicit
* tests: adjust tests
Co-authored-by: Charles Baynham <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Only reproduce steps once
* Add test
* Fix linter nasties
* Got the order wrong
* Check for previous run
* Formatting
* Do the de-duplication in _reproduce_stages
Co-authored-by: Charles Baynham <[email protected]>
* remote: locally index list of checksums available on cloud remotes
* fix review issues
- use pytest fixtures
- use sha256 digest
- s/fd/fobj/
- check more specific IO errors
* remote: use .dir checksums to validate index
* command: add --drop-index option for push/pull/status
* tests: --drop-index unit tests
* remote: make index threadsafe
* remote: force re-index after gc
* only save/dump index once per command
* remote: add helper index functions update_all/replace_all
* remote: store index as sqlite3 database
* fix deepsource warnings
* rename --drop-index to --clear-index
* rename index.invalidate to index.clear
* functional tests for remote.index
* fix index not being updated on status -c
* remote: add index.intersection()
* remote: use index when making assumptions about remote .dir contents
* add missing func tests
* do not index standalone files
* cleanup index test fixture
* fix deepsource warnings
* update autocompletion scripts/cli help message for --clear-index
* tests: remove unused mocker
* push: only index successfully pushed dirs
- do not index files on partial push/upload
* remote: revert behavior to trust .dir files on remote
- skip unnecessary index check for dir contents if .dir file exists on
the remote
* fix missing RemoteIndexNoop functions
* separate index validation from status dir exists query
* bugfix: include indexed checksums for all cache_exists cases
* review fix: cleanup debug messages
* tests: fix DS warning
* remove --clear-index
* remove unused index functions
Now, pipeline stage files can house multiple stages, and separate lockfiles
are created which has the checksums, whereas Dvcfile will be clean and human
readable and editable. The *.dvc files will be generated for output files.
It is available via a hidden flag: -n.
Fixes#1871
PR: #3584
Ruslan Kuprieiev
Peter Rowlands (변기호)
Shankho Boron GhoshandRestyled.io
Casper da Costa-Luis
Saugat Pachhai
Jack WellsandRuslan Kuprieiev
nik123
Paweł RedzyńskiandRuslan Kuprieiev
Ivan ShchekleinandJorge Orpinel
