* remote: move upload()/download() into tree
* remote: move path_info/path_cls into tree
* remote.azure: finish moving methods into tree
* remote.gs: finish moving methods into tree
* remote.hdfs: finish moving methods to tree
* remote.http: finish moving methods into tree
* remote.oss: finish moving methods into tree
* remote.s3: finish moving methods into tree
* remote.ssh: finish moving methods into tree
* remote.gdrive: finish moving methods into tree
* tests: update remote unit tests
* tests: update func tests
* remote: use walk_files() for all remotes
* list_cache_paths() now uses tree.walk_files() for all remotes except
local/ssh
* fix DS warnings
* bugfixes
* BaseRemote: start moving tree functions into BaseRemoteTree
* remote: add LocalRemoteTree
* remote: add cloud remote trees
* remote: move remove(), makedirs() into tree
* remote: move move(), copy(), symlink(), hardlink(), reflink() into tree
* fix attributes for moved remote methods
* tests: update unit tests for moved remote tree methods
* tests: mv unit/remote/test_remote_dir.py unit/remote/test_remote_tree.py
* fix attributes for moved remote tree functions
* tests: update func tests for moved remote tree methods
* test fixes
* fix DS warnings
`pipeline list` is no more, it had some very questionable applications
(none to be precise) and was duplicating old `pipeline show` functionality.
We might introduce a different command to assist in listing (e.g. for
shell completion) in the future.
New `dvc dag` replaces `dvc pipeline show` and by-default will show
all pipelines in the project in ASCII (so there is no more --ascii)
and also `--dot` support, just like before. When given a target,
`dvc dag` (in any mode) will show DAG that leads to the target,
skipping the descendants, unless `--full` is specified.
This should allow us to be able to remove a stage from a dvcfile,
(if it's a single stage file, it removes it), and make us able to
delete the overwrite the stage on run as well.
This commit:
* changes `run` `--overwrite-dvcfile` flag to `--overwrite`
* removes prompt in `run` for overwrite and fails hard
* changes `remove`: now it by default removes stage entry and unprotect outs
* `remove` has `--outs` to remove outputs instead of unprotecting them
* addressing: allow stage name addressing without colon
* repo: reorganize imports
* refactor/reorganize collect_granular
* add tests for stage addressing
* fix ds and cc issues
* fix unpack error in test
* tests: stage string representations
* exception: add error message
Also remove `cause` when throwing NoOutputOrStage, as
CheckoutErrorSuggestGit will start chaining all the messages
in cause of the exception.
* rename NoOutputOrStage NoOutputOrStageError
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* remote: remove unpacked dir
We've done a lot of optimizations lately, which made unpacked dir trick
obsoleted.
* local: save us a stat call in is_protected
* local: speedup checksum_to_path ~x5.5
* tree: walk: don't use os.path.join
It is ~5.5 times slower than joining by hand.
* dvcignore: optimize matching x10
On a repo with a dvcignore with 1 pattern and a directory with 400K
files, `dvc status` now takes ~8 sec instead of ~30 sec.
To achieve that, we make some assumptions about the paths formats that
we are dealing with, so we could use simpler logic instead of using
very slow `relpath`, `abspath` etc on every entry in a directory.
It is also clear that CleanTree behavior is inconsistent (even tests
expect very different outputs from it), so we will need to look into
this later.
* azure: support external dependencies and outputs
- Implement get_file_checksum with get_etag
- Implement a DependencyAzure and OutputAzure classes
- Add AzureOutput to OUTS, OUTS_MAP and CHECKSUMS_SCHEMA
- Add AzureDependency to DEPS and DEPS_MAP
- Add unit tests for AzureOutput and AzureDependency
Fixes#3540.
* Use content_md5 instead of etag
The etag is different for the same file in different locations.
* Use etag and remove external outputs for Azure
Due to inconsistencies with Azure's Content-MD5 property, it is not
stable enough to be used at the moment. Using the etag offers enough
reliability to support external dependencies. See also:
https://discordapp.com/channels/485586884165107732/565699007037571084/713462619469643796
* plots: remove --select from CLI
It needs reconsideration, we can't release it as is right now. E.g. it
expects xpath to start with `$`, but that will get interpreted by your
shell, which is bad.
* completion: update plots
* DvcTree: use out.dir_cache instead of out.collect_use_dir_cache
* RepoTree: support filtering out dvcfiles during walk()
* RepoTree: support recursive isdvc()
* remote: support computing checksums for tree objects
* RepoTree: pull dir contents on walk() if in `fetch` mode
* tests: add func test for DVC dir pull on RepoTree.walk()
* DvcTree: support granular open()
* RepoTree: add get_file_checksum()
* remote: revert checksum changes
* remote: add tree support for cache.save()
* tests: test saving directly to cache from RepoTree
* remote: local cache needs its own WorkingTree instance when repo.tree is
GitTree
* RepoTree: state context should be entered by open()/walk() caller
* RepoTree: only pull dir contents when cache has changed
* tree: update tree tests
* fix ds warnings, update comments
* fix windows tests
* Renamed --outs-only to --dvc-only
* Changed outs_only to dvc_only
* replace output_only with dvc_only
* Restyled by black
Co-authored-by: Restyled.io <[email protected]>
* run: add --plot/--plot-no-cache
Part of #3409
* plot: move show/diff to API
* plots: show marked plots by default
Involves some refactoring to make plots comply with the rest of the code
base.
* dvc.yaml: use top-level metrics and plots
* dvc.yaml: add support for plot template
* plots: add support for templates in dvc.yaml
* dvc.yaml: allow metrics/plots to have persist/cache
* plots: render single page by-default and introduce --show-json
* plot: use "working tree" instead of "workspace"
Just to sync this with the rest of the codebase. We should consider
renaming it to "workspace" everywhere, but for now I opt to not touch
it to not break something that depends on it.
* plots: rename -f|--file to -o|--out
Better corresponds to other commands like `import`.
* plots: use RepoTree and brancher instead of api
* stage: get rid of OutputParams
* tests: plots: add simple diff test
* plots: use `--targets` instead of `--datafile`
Makes it comply with `dvc metrics/params` CLI options.
* tests: unit: plots: add --show-json test
* stage: cleanup exception for format error
Based on feedback from the users.
* lockfile: use relpath in corruption error
* GitTree: add stat() function
* BaseTree: make walk_files a base tree method
* WorkingTree: add stat() method (wrapper for os.stat)
* utils.fs: support non-working tree for get_mtime_and_size
* utils.fs: add copy_fobj_to_file
* utils: support using tree.open in file_md5
* DvcTree: support stream or fetch on open()
* tests: test DvcTree dir cache support for walk()
* RepoTree: finish implementation of BaseTree functions
* repo: use RepoTree/DvcTree.open() for repo.open_by_relpath
* repo: only use RepoTree when needed for now
* fix docstring
* RepoTree: only use DvcTree.open for DVC outs
* dvc: rename plot to plots
Follows `dvc params/metrics` convention and is needed in preparation for new
commands and some refactoring.
Part of #3409
* completion: rename plot -> plots
* stage: extract unnecessary methods out of Stage class
* stage: change naming of functions
* stage: extract dump equal to utils
* stage: extract computation of md5 to utils
* stage: extract command run related logic in run.py
* stage: reorganize run()
* stage: boolean cosmetic changes
* stage: reorder save() and extract some into func
* stage: refactor status() into smaller functions
* extract resolve_wdir into utils
* merge _changed() and changed()
* cosmetic changes
* extract dump to utils
* extract import updates/sync to imports.py
* remove meta from PipelineStage
* fix imports, params load
* make run_stage a function, fix mocks in tests
* use original cmd
* rollback imports on decorators
* fix ds and cc
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
Currently we only restore `--outs`, but `--outs-no-cache` are also
useful, especially when they are metrics files
(`-M|--metrics-no-cache`). So this PR makes run-cache save uncached
files to our cache and also tries to checkout them back during the
`restore` phase.
* params/metrics: diff: implement tabulated markdown output
Via `--show-md` on `diff`s.
* table: show None on missing val
* tests: use `dedent`, add tests for markdown
* diff: add --show-md
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev
Alexander Schepanovski
Saugat Pachhai
Peter Rowlands (변기호)
Steffan Sluis
Shankho Boron GhoshandRestyled.io
Casper da Costa-Luis
Jack WellsandRuslan Kuprieiev
nik123