We will be re-introducing `metrics modify` in the future and
will likely revisit `add/remove` too, so it is better to get
rid of these for now and introduce later with a clear scneario
in mind.
Fixes#3849Fixes#3907
`pipeline list` is no more, it had some very questionable applications
(none to be precise) and was duplicating old `pipeline show` functionality.
We might introduce a different command to assist in listing (e.g. for
shell completion) in the future.
New `dvc dag` replaces `dvc pipeline show` and by-default will show
all pipelines in the project in ASCII (so there is no more --ascii)
and also `--dot` support, just like before. When given a target,
`dvc dag` (in any mode) will show DAG that leads to the target,
skipping the descendants, unless `--full` is specified.
* pre-commit: use local hooks
* pre-commit: don't run dvc in parallel
By default pre-commit will try to run hooks in parallel, which results
in some dvc instances failing to acquire repo locks.
Fixes#3751
* remote: remove unpacked dir
We've done a lot of optimizations lately, which made unpacked dir trick
obsoleted.
* local: save us a stat call in is_protected
* local: speedup checksum_to_path ~x5.5
* tree: walk: don't use os.path.join
It is ~5.5 times slower than joining by hand.
* dvcignore: optimize matching x10
On a repo with a dvcignore with 1 pattern and a directory with 400K
files, `dvc status` now takes ~8 sec instead of ~30 sec.
To achieve that, we make some assumptions about the paths formats that
we are dealing with, so we could use simpler logic instead of using
very slow `relpath`, `abspath` etc on every entry in a directory.
It is also clear that CleanTree behavior is inconsistent (even tests
expect very different outputs from it), so we will need to look into
this later.
* plots: remove --select from CLI
It needs reconsideration, we can't release it as is right now. E.g. it
expects xpath to start with `$`, but that will get interpreted by your
shell, which is bad.
* completion: update plots
* run: add --plot/--plot-no-cache
Part of #3409
* plot: move show/diff to API
* plots: show marked plots by default
Involves some refactoring to make plots comply with the rest of the code
base.
* dvc.yaml: use top-level metrics and plots
* dvc.yaml: add support for plot template
* plots: add support for templates in dvc.yaml
* dvc.yaml: allow metrics/plots to have persist/cache
* plots: render single page by-default and introduce --show-json
* plot: use "working tree" instead of "workspace"
Just to sync this with the rest of the codebase. We should consider
renaming it to "workspace" everywhere, but for now I opt to not touch
it to not break something that depends on it.
* plots: rename -f|--file to -o|--out
Better corresponds to other commands like `import`.
* plots: use RepoTree and brancher instead of api
* stage: get rid of OutputParams
* tests: plots: add simple diff test
* plots: use `--targets` instead of `--datafile`
Makes it comply with `dvc metrics/params` CLI options.
* tests: unit: plots: add --show-json test
* stage: cleanup exception for format error
Based on feedback from the users.
* lockfile: use relpath in corruption error
* dvc: rename plot to plots
Follows `dvc params/metrics` convention and is needed in preparation for new
commands and some refactoring.
Part of #3409
* completion: rename plot -> plots
Currently we only restore `--outs`, but `--outs-no-cache` are also
useful, especially when they are metrics files
(`-M|--metrics-no-cache`). So this PR makes run-cache save uncached
files to our cache and also tries to checkout them back during the
`restore` phase.
To be consistent with how git does it (e.g. objects) as well as with
future cache structure where we plan on adding `dirs`/`files`/`objects`
subdirs in the future.
Unlike old `_check_missing_deps`, this also verifies that we are able to
save more complex dependencies such as parameters, where we not only
care about the config file, but also about the parameters in it.
Fixes#3707
`get_checksum()` recomputes the checksum which might not match the
pre-recorded one. `checksum` is the one that was `save()`ed during run
and it is the one that should be used in the lockfile.
This patch introduces `.dvc/cache/stages` that is used to store previous
runs and their results, which could then be reused later when we stumble
upon the same command with the same deps and outs.
Format of build cache entries is single-line json, which is readable by
humans and might also be used for lock files discussed in #1871.
Related to #1871
Local part of #1234
Current implementation doesn't work with directories, which results in
hard-to-debug bugs like #3561. The bug that made us walk through index
manually is now fixed in gitpython, so we can simply use `ls_files` from
now on. Added some tests to prevent this from happening in the future.
Fixes#3561
This is useful as it is in `dvc ls` implementation, but this is also a
prerequisite for `RepoTree` that will combine git(workspace) files and
dvc files into one tree, allowing things like `dvc metrics` and
`external_repo` to have a universal interface to access whatever
file/dir they want without cumbersome git/dvc path detection.
Using it to load dirs takes around 50 sec compared to 0.4 with simple
`replace()`. Path objects are great, but not when you need lots of them.
Related to #3635
Current logic is extremely complex and messy, hiding a lot of bugs
inside. There are pretty much only 2 scenarios for stage creation:
1) `dvc add/import/etc` when dvcfile is created beside the output file
2) `dvc run` when dvcfile is created at the place of execution
both are transparently affected by `--fname` flag. Since we've removed
`--cwd` complexity already, there is no reason to generalize stage path
resolution for those two scenarios.
Windows (or rather NTFS) doesn't allow deleting read-only files, which made us
try to chmod +w first. But that is not necessery on Linux and Mac, and
has led to unnecessary cache file mode changes, which was causing issues
in shared cache scenario when a user that doesn't own a file was trying
to automatically protect it.
Fixes#3618
Texttable will try to detect the column type automatically, which
results in a low precision of the float that it is showing. We need to
disable that auto-formatting and just use `str(val)` instead.
Makes it easier to use in shell scripts. E.g. `-z $(dvc metrics diff)`, which
has the same semantics as `git diff`. Also syncs up with the behaviour
of `dvc diff`.
Fixes#3469
* remote: base: improve exception message
* remote: use `all()` instead of `list_cache_paths()`
`list_cache_paths()` doesn't ignore paths that don't look like hashes,
which makes dvc raise an exception when it tries to convert such paths
to hashes. We should use `all()` instead.
Discord context:
https://discordapp.com/channels/485586884165107732/485596304961962003/694454282719592458
* remote: all: print debug message when skipping paths
This will allow us to see it from verbose logs coming from users.
Ruslan Kuprieiev
