* stage: make runs checkpointable
* api: add make_checkpoint API call
* tests: add test for checkpointed stage
* experiments: implement checkpoints callback chain
* initial working state
* use .dvc/tmp for checkpoint signal file location
* force repro for checkpoint experiments
* checkout: add flag to allow missing persistent outputs
(i.e. not yet created)
* use allow_persist_missing on checkpoint experiment repro
* executor: force checkout before checkpoints experiment repro
* move checkpoint commands to `dvc exp run`
* support resuming checkpoint runs with --continue
* fix git bugs
* experiments: include checkpoint commits in `exp show`
* use 1sec sleep timers
* fix apply_workspace test conflict
* tests: add test for `dvc exp run`
* cleanup signal file behavior
* fix bool conversion
* tests: add tests for checkpoint and checkpoint_continue
* fix styling
* fix checkpoint stage test
* fix checkpoint monitor thread condition/notification
* repro: support --pull
This option allows run-cache to pull the cache for the outputs
when restoring a stage instead of pulling everything at once during
`dvc pull --run-cache`.
Fixes#4223
* repro: adjust --pull help
* metrics: diff: make table more intuitive
Currently we have a "diff not supported" marker in the table, that is
used whenever dvc is not able to diff the values (e.g. no old one, no
new one or format error). It is quite ugly and it is much more intuitive
to just show both old and new values and now show that marker at all, as
if there is no old/new value it is intuitive that we can't calculate the
change. Next step is to add old/new "bad format"/etc markers for
unsupported types or malformed/missing metric files.
E.g., before:
```
Path Metric Value Change
metrics.json foo 1.2 diff not supported
```
after this PR:
```
Path Metric Old New Change
metrics.json foo None 1.2
```
* metrics/params: use em dash in diff table
* diff: support missing cache status for workspace diff
* diff: add --hide-missing option to hide cache missing status
* update command tests
* update diff func tests
* only show missing cache status when diffing local workspace
* suppress all output if no changes and --hide-missing
* use hide_missing instead of show_missing
* use is_dvc
* experiments: cleanup show UI
* show git tag/branch names instead of SHAs by default
* make timestamp formatting more like the viewer
* update tests
* place timestamps in their own column
* update command test
* exp: remove unused regex
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* plots: add -e/--experiment option for plots diff
* experiments: add method to get baseline for a given experiment
* plots: default to diff against baseline instead of workspace/head for experiments
* use exact match for experiments RE since name-rev may return relative
commits
* diff: Add sort logic to sort diff ouput irrespective of output format(default/json/md)
* diff: test stub using actual value, not pass by reference for (dict)
* experiments: add initial local tmpdir executor
* fix detached head pull
* experiments: include unchanged (unrepro'd) stages in experiment hash
* experiments: repro single experiment using tmpdir executor by default
* stage experiments as stash commits before running
* include repro args/kwargs in stashed experiments
* support running arbitrary experiment commits (including stash commits)
* experiments: use ProcessPoolExecutor
* experiments: add `dvc exp` alias for `dvc experiments`
* experiments: add `dvc repro --experiment --queue`
* `--queue` can be used to stage an experiment for future execution
* experiments: show queued (unexecuted) experiments in `dvc exp show`
* revert ProcessPoolExecutor change
* experiments: add `dvc repro --run-all [--jobs]`
* `--run-all` can be used to run all queued experiments in parallel
* cleanup output
* experiments: use ProcessPoolExecutor
* fix returning unpicklable objects error
* update tests
* fix windows cleanup issue
* on windows tempdir cannot be removed if we are chdir'd into that
directory
* experiments: add initial experiments.show
* only includes metrics + params (no code/data information)
* experiments: basic `dvc experiments show` functionality
* experiments: clone and run repro inside clone workspace
* repro: add -e/--experiment option
* experiments: hash experiments to identify duplicates
* experiments: update show
* experiments: checkout experiment after running repro
* experiments: add `dvc experiments checkout`
* experiments: add simple `dvc experiments diff` command
* just shows combined output from `metrics diff` and `plots diff` for
now
* add very simple tests
* add some useful output messages
* experiments: suppress help messages while feature is in development
* experiment related commands will be accessible but hidden from the
default command help messages
* add simple experiments command tests
* experiments: disable feature by default
* experiments only enabled in test environment (DVC_TEST) or when
core.experiments config option is true
* use fetch on checkout missing revs
Also, makes fixes for `--overwrite` rename to `--force`
and also remove `remove_with_prompt` which was only being
used with `force\=True` anyway.
Fix#4003
`pipeline list` is no more, it had some very questionable applications
(none to be precise) and was duplicating old `pipeline show` functionality.
We might introduce a different command to assist in listing (e.g. for
shell completion) in the future.
New `dvc dag` replaces `dvc pipeline show` and by-default will show
all pipelines in the project in ASCII (so there is no more --ascii)
and also `--dot` support, just like before. When given a target,
`dvc dag` (in any mode) will show DAG that leads to the target,
skipping the descendants, unless `--full` is specified.
This should allow us to be able to remove a stage from a dvcfile,
(if it's a single stage file, it removes it), and make us able to
delete the overwrite the stage on run as well.
This commit:
* changes `run` `--overwrite-dvcfile` flag to `--overwrite`
* removes prompt in `run` for overwrite and fails hard
* changes `remove`: now it by default removes stage entry and unprotect outs
* `remove` has `--outs` to remove outputs instead of unprotecting them
* plots: remove --select from CLI
It needs reconsideration, we can't release it as is right now. E.g. it
expects xpath to start with `$`, but that will get interpreted by your
shell, which is bad.
* completion: update plots
* Renamed --outs-only to --dvc-only
* Changed outs_only to dvc_only
* replace output_only with dvc_only
* Restyled by black
Co-authored-by: Restyled.io <[email protected]>
* run: add --plot/--plot-no-cache
Part of #3409
* plot: move show/diff to API
* plots: show marked plots by default
Involves some refactoring to make plots comply with the rest of the code
base.
* dvc.yaml: use top-level metrics and plots
* dvc.yaml: add support for plot template
* plots: add support for templates in dvc.yaml
* dvc.yaml: allow metrics/plots to have persist/cache
* plots: render single page by-default and introduce --show-json
* plot: use "working tree" instead of "workspace"
Just to sync this with the rest of the codebase. We should consider
renaming it to "workspace" everywhere, but for now I opt to not touch
it to not break something that depends on it.
* plots: rename -f|--file to -o|--out
Better corresponds to other commands like `import`.
* plots: use RepoTree and brancher instead of api
* stage: get rid of OutputParams
* tests: plots: add simple diff test
* plots: use `--targets` instead of `--datafile`
Makes it comply with `dvc metrics/params` CLI options.
* tests: unit: plots: add --show-json test
* stage: cleanup exception for format error
Based on feedback from the users.
* lockfile: use relpath in corruption error
* dvc: rename plot to plots
Follows `dvc params/metrics` convention and is needed in preparation for new
commands and some refactoring.
Part of #3409
* completion: rename plot -> plots
* params/metrics: diff: implement tabulated markdown output
Via `--show-md` on `diff`s.
* table: show None on missing val
* tests: use `dedent`, add tests for markdown
* diff: add --show-md
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Note: This does not change default for `pipeline` as
it cannot show multiple DAGs at the same time. If we
are able to show multiple DAGs, we can just default to
`dvc.yaml`.
* pull/fetch/push: add stats
* fix typo: s/logging/logger
* pull: make stats detailed, with summary at the bottom
* tests: mock properly, set return_value
* add a line space
Now, pipeline stage files can house multiple stages, and separate lockfiles
are created which has the checksums, whereas Dvcfile will be clean and human
readable and editable. The *.dvc files will be generated for output files.
It is available via a hidden flag: -n.
Fixes#1871
PR: #3584
* refactor stage into dvcfile
* Split create_stage out of dvcfile to Stage::create() and overwrite
* Remove logging from Repo::run
* tests: stop using --remove-outs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Peter Rowlands (변기호)
Ruslan Kuprieiev
Raymond GasperandRuslan Kuprieiev
Pavel DitenbirandRuslan Kuprieiev
Puneetha Pai
Saugat Pachhai
karajan1001andRuslan Kuprieiev
Alexander Schepanovski
Shankho Boron GhoshandRestyled.io
nik123
Casper da Costa-Luis
Paweł RedzyńskiandRuslan Kuprieiev