* tests/remote: adding tests for real oss
the new test suggests #4633 is a bug in the emulator.
* Update tests/func/test_data_cloud.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update tests/remotes/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tests: correcting small lint issue
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* command: print the current cache directory when no cmd flags are provided
tests: corrected the test for cache dir cmomand
comamnd 'dvc cache dir' now prints the current cache dir
when no cmd flags are provided and modified existing test for
the same.
Fixes#1998
Signed-off-by: Girish Joshi <[email protected]>
* command: updated validation for args.value and args.unset flag
according to current convention in dvc/command/cache.py
Signed-off-by: Girish Joshi <[email protected]>
* Update dvc/command/cache.py
* dont forget logger
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
When executing `dvc import-url remote://some-webdav/data`, the name of
the WebDAV dependency in the created stage for data used to be `None`.
This resulted in the following error:
```
Traceback (most recent call last):
File "/home/schubert/projects/dvc/dvc/main.py", line 76, in main
ret = cmd.run()
File "/home/schubert/projects/dvc/dvc/command/imp_url.py", line 14, in run
self.repo.imp_url(
File "/home/schubert/projects/dvc/dvc/repo/__init__.py", line 51, in wrapper
return f(repo, *args, **kwargs)
File "/home/schubert/projects/dvc/dvc/repo/scm_context.py", line 4, in run
result = method(repo, *args, **kw)
File "/home/schubert/projects/dvc/dvc/repo/imp_url.py", line 54, in imp_url
stage.run()
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/site-packages/funcy/decorators.py", line 39, in wrapper
return deco(call, *dargs, **dkwargs)
File "/home/schubert/projects/dvc/dvc/stage/decorators.py", line 36, in rwlocked
return call()
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/site-packages/funcy/decorators.py", line 60, in __call__
return self._func(*self._args, **self._kwargs)
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 450, in run
self.save()
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 395, in save
self.md5 = self.compute_md5()
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 388, in compute_md5
m = compute_md5(self)
File "/home/schubert/projects/dvc/dvc/stage/utils.py", line 167, in compute_md5
return dict_md5(
File "/home/schubert/projects/dvc/dvc/utils/__init__.py", line 118, in dict_md5
return dict_hash(d, "md5", **kwargs)
File "/home/schubert/projects/dvc/dvc/utils/__init__.py", line 113, in dict_hash
byts = json.dumps(filtered, sort_keys=True).encode("utf-8")
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
TypeError: '<' not supported between instances of 'str' and 'NoneType'
```
* state: add simple reentrance
* state: pass Repo instead of LocalCache
* state: use granularly
Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
* stage: make runs checkpointable
* api: add make_checkpoint API call
* tests: add test for checkpointed stage
* experiments: implement checkpoints callback chain
* initial working state
* use .dvc/tmp for checkpoint signal file location
* force repro for checkpoint experiments
* checkout: add flag to allow missing persistent outputs
(i.e. not yet created)
* use allow_persist_missing on checkpoint experiment repro
* executor: force checkout before checkpoints experiment repro
* move checkpoint commands to `dvc exp run`
* support resuming checkpoint runs with --continue
* fix git bugs
* experiments: include checkpoint commits in `exp show`
* use 1sec sleep timers
* fix apply_workspace test conflict
* tests: add test for `dvc exp run`
* cleanup signal file behavior
* fix bool conversion
* tests: add tests for checkpoint and checkpoint_continue
* fix styling
* fix checkpoint stage test
* fix checkpoint monitor thread condition/notification
* repro: support --pull
This option allows run-cache to pull the cache for the outputs
when restoring a stage instead of pulling everything at once during
`dvc pull --run-cache`.
Fixes#4223
* repro: adjust --pull help
* yaml: wrap ruamel's DuplicateKeyError
Unfortunately ruamel's exception doesn't contain much info that we could
reuse, so we can only reasonably extract the exc.problem and have
something like:
```
ERROR: failed to obtain data status - unable to read: 'dvc.yaml', found
duplicate key "metrics" with value "[]" (original value: "[]")
```
Fixes#4664
* status: remove unneeded error prefix
* metrics: diff: make table more intuitive
Currently we have a "diff not supported" marker in the table, that is
used whenever dvc is not able to diff the values (e.g. no old one, no
new one or format error). It is quite ugly and it is much more intuitive
to just show both old and new values and now show that marker at all, as
if there is no old/new value it is intuitive that we can't calculate the
change. Next step is to add old/new "bad format"/etc markers for
unsupported types or malformed/missing metric files.
E.g., before:
```
Path Metric Value Change
metrics.json foo 1.2 diff not supported
```
after this PR:
```
Path Metric Old New Change
metrics.json foo None 1.2
```
* metrics/params: use em dash in diff table
* dvctree: use string paths for performance reasons in
_get_granular_checksum()
* dvctree: cache most recently accessed metadata
* dvctree: cache output dir cache during walk() calls
* diff: use get_file_hash directly instead of get_hash when appropriate
* dvctree: cache most recen dir_cache hashes in _get_granular_checksum
* revert metadata cache changes
- mostly unnecessary after granular checksum changes, will be better
addressed by the future exception handling updates
* fix windows path issue
* HashInfo: store HashInfo.dir_info as a dict instead of list
* DvcTree: use hash_info.dir_info for getting granular file hashes
* use str relpath
* use correct dir_info hash type
* fix windows paths
* Run CI for Python3.9, skips hdfs tests for 3.9
Also, `dvc[hdfs]` install will be a no-op till we have
a pyarrow wheel for 3.9.
Note that this does not add an entry to the classifier
claiming it supports 3.9, that will be added when 3.9
gets released (or, wait for pyarrow to be released).
* Skip HDFS related tests on Python3.9
Also, fix the python version regex for `3.9.0b5` like versions.
* Exclude python3.9 for Windows
libxml does not have wheels for Python39.
We do compile them on Linux and Mac
* Update tests/unit/test_info.py
* Use realpath for gitignore
Windows tests for Python3.8 was failing
intermittently.
Note: Also removed the realpath wrap around self.root_dir as it's already a
realpath when `Git` is instantiated.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Juliano Efson SalesandRuslan Kuprieiev
Jonny Browningandbrowningjp
Paweł Redzyński
Peter Rowlands (변기호)
Saugat Pachhai
Jorge OrpinelandRuslan Kuprieiev
omar.zhouandzwh
Frederik Schubert
Guillermo Blanco
Casper da Costa-Luis