* stage: make runs checkpointable
* api: add make_checkpoint API call
* tests: add test for checkpointed stage
* experiments: implement checkpoints callback chain
* initial working state
* use .dvc/tmp for checkpoint signal file location
* force repro for checkpoint experiments
* checkout: add flag to allow missing persistent outputs
(i.e. not yet created)
* use allow_persist_missing on checkpoint experiment repro
* executor: force checkout before checkpoints experiment repro
* move checkpoint commands to `dvc exp run`
* support resuming checkpoint runs with --continue
* fix git bugs
* experiments: include checkpoint commits in `exp show`
* use 1sec sleep timers
* fix apply_workspace test conflict
* tests: add test for `dvc exp run`
* cleanup signal file behavior
* fix bool conversion
* tests: add tests for checkpoint and checkpoint_continue
* fix styling
* fix checkpoint stage test
* fix checkpoint monitor thread condition/notification
* repro: support --pull
This option allows run-cache to pull the cache for the outputs
when restoring a stage instead of pulling everything at once during
`dvc pull --run-cache`.
Fixes#4223
* repro: adjust --pull help
* metrics: diff: make table more intuitive
Currently we have a "diff not supported" marker in the table, that is
used whenever dvc is not able to diff the values (e.g. no old one, no
new one or format error). It is quite ugly and it is much more intuitive
to just show both old and new values and now show that marker at all, as
if there is no old/new value it is intuitive that we can't calculate the
change. Next step is to add old/new "bad format"/etc markers for
unsupported types or malformed/missing metric files.
E.g., before:
```
Path Metric Value Change
metrics.json foo 1.2 diff not supported
```
after this PR:
```
Path Metric Old New Change
metrics.json foo None 1.2
```
* metrics/params: use em dash in diff table
* Run CI for Python3.9, skips hdfs tests for 3.9
Also, `dvc[hdfs]` install will be a no-op till we have
a pyarrow wheel for 3.9.
Note that this does not add an entry to the classifier
claiming it supports 3.9, that will be added when 3.9
gets released (or, wait for pyarrow to be released).
* Skip HDFS related tests on Python3.9
Also, fix the python version regex for `3.9.0b5` like versions.
* Exclude python3.9 for Windows
libxml does not have wheels for Python39.
We do compile them on Linux and Mac
* Update tests/unit/test_info.py
* Use realpath for gitignore
Windows tests for Python3.8 was failing
intermittently.
Note: Also removed the realpath wrap around self.root_dir as it's already a
realpath when `Git` is instantiated.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Fix error during external outputs checkout
User in discord complained that external outputs' checkout
was failing. Looks like, during refactor of `HashInfo`,
this was forgotten, and as no tests existed before, we had
a regression from 1.6.5 onwards.
* fix tests
* Restyled by black (#4571)
Co-authored-by: Restyled.io <[email protected]>
Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>
Co-authored-by: Restyled.io <[email protected]>
* `check-ignore` show sub-repo patterns
fix#4555
1. Sub-repo pattern now would be considered in `check-ignore` command.
2. New outputs for sub-repo patterns in a `detail`
3. Related tests.
* message change
* Print dvc version info for debugging
fix#4095
1. seperate info from dvc version.
2. print dvc version info when some unexpected error occured.
* Psutil NoneType error
* Update tests/func/test_version.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Move `package` outside dvc/utils/pkg.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
The API of `Dvcfile.dump` has been changed to
`dump(stage, update_pipeline=True, update_lock=True)`,
which will make it to always dump by default to both files,
and provide flags to suppress one or the other.
Fixes#4235
* diff: support missing cache status for workspace diff
* diff: add --hide-missing option to hide cache missing status
* update command tests
* update diff func tests
* only show missing cache status when diffing local workspace
* suppress all output if no changes and --hide-missing
* use hide_missing instead of show_missing
* use is_dvc
* diff: use RepoTree to get granular results for dir outs
* update tests for dir file granularity
* diff: prefer repo_tree.get_hash when possible
* DvcTree: handle extra files in dirty dvc out dirs
* support subrepos
This adds support for get/list/api/import/ls on a subrepo.
Also fixes#3180, by adding a granular url support for `dvc.api.get_url`
And, of course it fixes#3369.
* Add tests
* Fix tests, started to get lock_error
* Fix tests for windows
* try to fix mac builds
* Fix comments, readd removed comments
* Fix test
* address review suggestions:
* use `spy` instead of `wraps`
* remove dvc fixture in a test
* set `cache_types` on Git repo
By itself `self.info` is quite confusing, as it is not clear what it is
about. Using `hash_info` makes much more sense and is required to
support alternative hash types.
Related to #4144, #3069, #1676
fix#4482
According to the discussion. Make `DVC status [outputs]` output format the same with `DVC status [stages]` instead of `DVC status -c [outputs]`
* experiments: cleanup show UI
* show git tag/branch names instead of SHAs by default
* make timestamp formatting more like the viewer
* update tests
* place timestamps in their own column
* update command test
* exp: remove unused regex
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* plots: add -e/--experiment option for plots diff
* experiments: add method to get baseline for a given experiment
* plots: default to diff against baseline instead of workspace/head for experiments
* use exact match for experiments RE since name-rev may return relative
commits
* Speed up dvc files load
See `._load()` implementation. Alternative is not using 3 io calls, but
simply use open and catch exceptions.
Reading is slow for old git revisions.
* tree: git: cache tree and objects
* external: ignore PermissionError on Windows
This is the same thing we have in tests.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Outputs as target supporting for `dvc status`
fix#4191
1. Add a related test which would fail on current version.
* Pass the tests
1. add deps to the tests.
2. make change to pass the tests.
* Update tests/func/test_status.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Solve some change request.
Co-authored-by: Saugat Pachhai <[email protected]>
Ruslan Kuprieiev
Peter Rowlands (변기호)
Saugat PachhaiandRuslan Kuprieiev
Raymond GasperandRuslan Kuprieiev
Robert Lowry
Alexanderandaandrusenko
Paweł Redzyński
karajan1001
Alexander Schepanovski
nik123