* stage: make runs checkpointable
* api: add make_checkpoint API call
* tests: add test for checkpointed stage
* experiments: implement checkpoints callback chain
* initial working state
* use .dvc/tmp for checkpoint signal file location
* force repro for checkpoint experiments
* checkout: add flag to allow missing persistent outputs
(i.e. not yet created)
* use allow_persist_missing on checkpoint experiment repro
* executor: force checkout before checkpoints experiment repro
* move checkpoint commands to `dvc exp run`
* support resuming checkpoint runs with --continue
* fix git bugs
* experiments: include checkpoint commits in `exp show`
* use 1sec sleep timers
* fix apply_workspace test conflict
* tests: add test for `dvc exp run`
* cleanup signal file behavior
* fix bool conversion
* tests: add tests for checkpoint and checkpoint_continue
* fix styling
* fix checkpoint stage test
* fix checkpoint monitor thread condition/notification
* repro: support --pull
This option allows run-cache to pull the cache for the outputs
when restoring a stage instead of pulling everything at once during
`dvc pull --run-cache`.
Fixes#4223
* repro: adjust --pull help
* yaml: wrap ruamel's DuplicateKeyError
Unfortunately ruamel's exception doesn't contain much info that we could
reuse, so we can only reasonably extract the exc.problem and have
something like:
```
ERROR: failed to obtain data status - unable to read: 'dvc.yaml', found
duplicate key "metrics" with value "[]" (original value: "[]")
```
Fixes#4664
* status: remove unneeded error prefix
* metrics: diff: make table more intuitive
Currently we have a "diff not supported" marker in the table, that is
used whenever dvc is not able to diff the values (e.g. no old one, no
new one or format error). It is quite ugly and it is much more intuitive
to just show both old and new values and now show that marker at all, as
if there is no old/new value it is intuitive that we can't calculate the
change. Next step is to add old/new "bad format"/etc markers for
unsupported types or malformed/missing metric files.
E.g., before:
```
Path Metric Value Change
metrics.json foo 1.2 diff not supported
```
after this PR:
```
Path Metric Old New Change
metrics.json foo None 1.2
```
* metrics/params: use em dash in diff table
* dvctree: use string paths for performance reasons in
_get_granular_checksum()
* dvctree: cache most recently accessed metadata
* dvctree: cache output dir cache during walk() calls
* diff: use get_file_hash directly instead of get_hash when appropriate
* dvctree: cache most recen dir_cache hashes in _get_granular_checksum
* revert metadata cache changes
- mostly unnecessary after granular checksum changes, will be better
addressed by the future exception handling updates
* fix windows path issue
* HashInfo: store HashInfo.dir_info as a dict instead of list
* DvcTree: use hash_info.dir_info for getting granular file hashes
* use str relpath
* use correct dir_info hash type
* fix windows paths
* Run CI for Python3.9, skips hdfs tests for 3.9
Also, `dvc[hdfs]` install will be a no-op till we have
a pyarrow wheel for 3.9.
Note that this does not add an entry to the classifier
claiming it supports 3.9, that will be added when 3.9
gets released (or, wait for pyarrow to be released).
* Skip HDFS related tests on Python3.9
Also, fix the python version regex for `3.9.0b5` like versions.
* Exclude python3.9 for Windows
libxml does not have wheels for Python39.
We do compile them on Linux and Mac
* Update tests/unit/test_info.py
* Use realpath for gitignore
Windows tests for Python3.8 was failing
intermittently.
Note: Also removed the realpath wrap around self.root_dir as it's already a
realpath when `Git` is instantiated.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* status: Show friendly message for empty project.
When there are no stages in the repo, show a friendly message to that
effect and display a getting-started URL.
Fixes#4395.
* Fix style check failing due to line too long. (E501)
* Use format_link(…) as suggested by @skshetry, vs {blue}/{nc}.
Since I'd pulled the old syntax out of repo/init.py, I've fixed that up
as well so no one makes that mistake again. Note that this adds <>s around
the link text, unlike before:
+---------------------------------------------------------------------+
| |
| DVC has enabled anonymous aggregate usage analytics. |
| Read the analytics documentation (and how to opt-out) here: |
| <https://dvc.org/doc/user-guide/analytics> |
| |
+---------------------------------------------------------------------+
This is probably fine, since if we're going to have a standard link format
we might as well stick with it consistently. If we don't like it, we can
change format_link() I guess.
* Update dvc/command/status.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Paramiko has recently added support for `Match`
https://github.com/paramiko/paramiko/issues/717
which resulted in some users getting an import error about `invoke`
module, as it is not a default dependecy for paramiko.
@Pseudomanifold reported that installing `invoke` #4589 worked for
him, which means that this newly added paramiko feature works in
his case. That feature seems to not affect configs without `Match`,
so we could give this a try and install it by default.
Our conda package will be updated during the next release.
Fixes#4589Fixes#4616
Paweł Redzyński
Ruslan Kuprieiev
Peter Rowlands (변기호)
Guillermo Blanco
Saugat PachhaiandRuslan Kuprieiev
Casper da Costa-Luis
Raymond GasperandRuslan Kuprieiev
gsvoltandRestyled.io
Pavel DitenbirandRuslan Kuprieiev
Guillaume COTER
David RobertsandRuslan Kuprieiev
Robert Lowry
Alexanderandaandrusenko
David Roberts