* Allow setting of list parameteres
Check if dict is Mapping or list before using .get()
* Added test for dvc exp run with list paramater
* Add more tests for string parameter modification
* Escape file paths for calculation of md5 hashes on ssh remote
* Add tests for hardlink and improved md5 test with PropertyMock
* Add additional special characters to check for fix of issue #4628 and fix formatting
* Fix linting errors
* Move tests to correct file and use injected ssh_connection instead of setting it up from scratch
* Add test, and fix resolving of paths for params
Improve merging of vars and imported params, improve tracking
Resolve wdir correctly
* Fix windows relpath
* isort imports
* Fixes#4637
* use PathInfo instead of Path
* catch ValueError with PathInfo.relative_to and fallback to dvc.utils.relpath
* cleanup
* add test
Co-authored-by: Peter Rowlands <[email protected]>
* repo: make Repo.scm a cached attribute
* checkpoints: create final checkpoint after repro when needed
* use correct exception rev in _collect_and_commit
* command: print the current cache directory when no cmd flags are provided
tests: corrected the test for cache dir cmomand
comamnd 'dvc cache dir' now prints the current cache dir
when no cmd flags are provided and modified existing test for
the same.
Fixes#1998
Signed-off-by: Girish Joshi <[email protected]>
* command: updated validation for args.value and args.unset flag
according to current convention in dvc/command/cache.py
Signed-off-by: Girish Joshi <[email protected]>
* Update dvc/command/cache.py
* dont forget logger
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
When executing `dvc import-url remote://some-webdav/data`, the name of
the WebDAV dependency in the created stage for data used to be `None`.
This resulted in the following error:
```
Traceback (most recent call last):
File "/home/schubert/projects/dvc/dvc/main.py", line 76, in main
ret = cmd.run()
File "/home/schubert/projects/dvc/dvc/command/imp_url.py", line 14, in run
self.repo.imp_url(
File "/home/schubert/projects/dvc/dvc/repo/__init__.py", line 51, in wrapper
return f(repo, *args, **kwargs)
File "/home/schubert/projects/dvc/dvc/repo/scm_context.py", line 4, in run
result = method(repo, *args, **kw)
File "/home/schubert/projects/dvc/dvc/repo/imp_url.py", line 54, in imp_url
stage.run()
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/site-packages/funcy/decorators.py", line 39, in wrapper
return deco(call, *dargs, **dkwargs)
File "/home/schubert/projects/dvc/dvc/stage/decorators.py", line 36, in rwlocked
return call()
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/site-packages/funcy/decorators.py", line 60, in __call__
return self._func(*self._args, **self._kwargs)
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 450, in run
self.save()
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 395, in save
self.md5 = self.compute_md5()
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 388, in compute_md5
m = compute_md5(self)
File "/home/schubert/projects/dvc/dvc/stage/utils.py", line 167, in compute_md5
return dict_md5(
File "/home/schubert/projects/dvc/dvc/utils/__init__.py", line 118, in dict_md5
return dict_hash(d, "md5", **kwargs)
File "/home/schubert/projects/dvc/dvc/utils/__init__.py", line 113, in dict_hash
byts = json.dumps(filtered, sort_keys=True).encode("utf-8")
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
TypeError: '<' not supported between instances of 'str' and 'NoneType'
```
* state: add simple reentrance
* state: pass Repo instead of LocalCache
* state: use granularly
Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
* stage: make runs checkpointable
* api: add make_checkpoint API call
* tests: add test for checkpointed stage
* experiments: implement checkpoints callback chain
* initial working state
* use .dvc/tmp for checkpoint signal file location
* force repro for checkpoint experiments
* checkout: add flag to allow missing persistent outputs
(i.e. not yet created)
* use allow_persist_missing on checkpoint experiment repro
* executor: force checkout before checkpoints experiment repro
* move checkpoint commands to `dvc exp run`
* support resuming checkpoint runs with --continue
* fix git bugs
* experiments: include checkpoint commits in `exp show`
* use 1sec sleep timers
* fix apply_workspace test conflict
* tests: add test for `dvc exp run`
* cleanup signal file behavior
* fix bool conversion
* tests: add tests for checkpoint and checkpoint_continue
* fix styling
* fix checkpoint stage test
* fix checkpoint monitor thread condition/notification
* repro: support --pull
This option allows run-cache to pull the cache for the outputs
when restoring a stage instead of pulling everything at once during
`dvc pull --run-cache`.
Fixes#4223
* repro: adjust --pull help
* yaml: wrap ruamel's DuplicateKeyError
Unfortunately ruamel's exception doesn't contain much info that we could
reuse, so we can only reasonably extract the exc.problem and have
something like:
```
ERROR: failed to obtain data status - unable to read: 'dvc.yaml', found
duplicate key "metrics" with value "[]" (original value: "[]")
```
Fixes#4664
* status: remove unneeded error prefix
* metrics: diff: make table more intuitive
Currently we have a "diff not supported" marker in the table, that is
used whenever dvc is not able to diff the values (e.g. no old one, no
new one or format error). It is quite ugly and it is much more intuitive
to just show both old and new values and now show that marker at all, as
if there is no old/new value it is intuitive that we can't calculate the
change. Next step is to add old/new "bad format"/etc markers for
unsupported types or malformed/missing metric files.
E.g., before:
```
Path Metric Value Change
metrics.json foo 1.2 diff not supported
```
after this PR:
```
Path Metric Old New Change
metrics.json foo None 1.2
```
* metrics/params: use em dash in diff table
Peter Rowlands (변기호)
Paweł Redzyński
Tomás Farías Santana
Sami Jawhar
Samuel Neugber
Saugat Pachhai
Harsh Patel
Alexei Tenitski
Musa AtlıhanandPeter Rowlands
Jonny Browningandbrowningjp
Jorge OrpinelandRuslan Kuprieiev
omar.zhouandzwh
Frederik Schubert