* BaseRemote: start moving tree functions into BaseRemoteTree
* remote: add LocalRemoteTree
* remote: add cloud remote trees
* remote: move remove(), makedirs() into tree
* remote: move move(), copy(), symlink(), hardlink(), reflink() into tree
* fix attributes for moved remote methods
* tests: update unit tests for moved remote tree methods
* tests: mv unit/remote/test_remote_dir.py unit/remote/test_remote_tree.py
* fix attributes for moved remote tree functions
* tests: update func tests for moved remote tree methods
* test fixes
* fix DS warnings
We will be re-introducing `metrics modify` in the future and
will likely revisit `add/remove` too, so it is better to get
rid of these for now and introduce later with a clear scneario
in mind.
Fixes#3849Fixes#3907
`pipeline list` is no more, it had some very questionable applications
(none to be precise) and was duplicating old `pipeline show` functionality.
We might introduce a different command to assist in listing (e.g. for
shell completion) in the future.
New `dvc dag` replaces `dvc pipeline show` and by-default will show
all pipelines in the project in ASCII (so there is no more --ascii)
and also `--dot` support, just like before. When given a target,
`dvc dag` (in any mode) will show DAG that leads to the target,
skipping the descendants, unless `--full` is specified.
This should allow us to be able to remove a stage from a dvcfile,
(if it's a single stage file, it removes it), and make us able to
delete the overwrite the stage on run as well.
This commit:
* changes `run` `--overwrite-dvcfile` flag to `--overwrite`
* removes prompt in `run` for overwrite and fails hard
* changes `remove`: now it by default removes stage entry and unprotect outs
* `remove` has `--outs` to remove outputs instead of unprotecting them
* pre-commit: use local hooks
* pre-commit: don't run dvc in parallel
By default pre-commit will try to run hooks in parallel, which results
in some dvc instances failing to acquire repo locks.
Fixes#3751
* ignore: dvcignore root is distinct from tree_root
* for subrepos, dvcignore root should be the subrepo's .dvcignore, not
host repo's .dvcignore
* tests: add test for subrepo dvcignore behavior
* tests: add test for metrics show -T with pre-existing tags
* brancher: ignore git revisions that do not contain repo
* tests: update test to include cmd
* CleanTree: tree_root for subrepos should be the repo root, not scm root
* only force subrepo root for git trees
* fix windows shell quotes
* tests: test that CleanTree works outside of walk()
* ignore: CleanTree should always check ignores, not just for walk()
* CleanTree: handle broken symlinks
* handle case for linking from shared local cache dirs
* Fix windows relpath/different drive issue
* addressing: allow stage name addressing without colon
* repo: reorganize imports
* refactor/reorganize collect_granular
* add tests for stage addressing
* fix ds and cc issues
* fix unpack error in test
* tests: stage string representations
* exception: add error message
Also remove `cause` when throwing NoOutputOrStage, as
CheckoutErrorSuggestGit will start chaining all the messages
in cause of the exception.
* rename NoOutputOrStage NoOutputOrStageError
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* remote: remove unpacked dir
We've done a lot of optimizations lately, which made unpacked dir trick
obsoleted.
* local: save us a stat call in is_protected
* local: speedup checksum_to_path ~x5.5
* tree: walk: don't use os.path.join
It is ~5.5 times slower than joining by hand.
* dvcignore: optimize matching x10
On a repo with a dvcignore with 1 pattern and a directory with 400K
files, `dvc status` now takes ~8 sec instead of ~30 sec.
To achieve that, we make some assumptions about the paths formats that
we are dealing with, so we could use simpler logic instead of using
very slow `relpath`, `abspath` etc on every entry in a directory.
It is also clear that CleanTree behavior is inconsistent (even tests
expect very different outputs from it), so we will need to look into
this later.
* azure: support external dependencies and outputs
- Implement get_file_checksum with get_etag
- Implement a DependencyAzure and OutputAzure classes
- Add AzureOutput to OUTS, OUTS_MAP and CHECKSUMS_SCHEMA
- Add AzureDependency to DEPS and DEPS_MAP
- Add unit tests for AzureOutput and AzureDependency
Fixes#3540.
* Use content_md5 instead of etag
The etag is different for the same file in different locations.
* Use etag and remove external outputs for Azure
Due to inconsistencies with Azure's Content-MD5 property, it is not
stable enough to be used at the moment. Using the etag offers enough
reliability to support external dependencies. See also:
https://discordapp.com/channels/485586884165107732/565699007037571084/713462619469643796
Ruslan Kuprieiev
Alexander Schepanovski
Peter Rowlands (변기호)
Saugat Pachhai
Jorge Orpinel
Nicholas A. Del Grosso
karajan1001
Casper da Costa-Luis
Steffan Sluis
Paweł Redzyński