* dvc: include cache urls in info output.
Show cache urls for external and remote cache in the info output
to assist in diagnosis of issues.
Fixes#4574.
* Remove unnecessary constant check for generator expression.
* Add remotes and caches in version command for better diagnosis.
* Fix flaky unit test to be runtime independent by using regex search.
* Use Cache.by_scheme() for enumerating supported caches.
* Add get_cloud_schemes routine for retrieval of configured remote schemes.
Add tests for local remote and remote:// scheme for info.
* Cleanup unused code.
* info: refactor remote parsing
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Add support for WebHDFS
* Style and linting fixes
* Implement abstract methods and disable no-member pylint errors
* Add init unit test
* Yield client and fix redefinition of hdfs
* Get dynamic hdfs Docker port
* Add support for open in WebHDFSTree
* Remove hdfscli_config fallbacks in WebHDFSTree
* Use port 50070 for webhdfs
* WebHDFSTree supports http URL and proper walking
* Fix failing unit test after refactoring
* Add overwrite argument to upload
* Add remaining func tests for webhdfs
* Remove print statements
* Allow dynamic attributes in pylint for hdfs library
* Comment error on instantiating HDFS client from config
* Remove depth argument from call to walk
* Update dvc/tree/webhdfs.py
* Remove unnecessary deque
* Check exception messages hdfs client failure from config
* Return client directly
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Split long line
* Add size to HashInfo
* Implement move via hdfs.rename
* Add missing line
* Make parent directories before moving file in WebHDFS
* Add total to progress bar
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tree: use file_mode in upload() for trees that support modes
* experiments: preserve file_mode for files in the workspace
* skip permissions tests on windows
* LocalTree: separate chmod from protect
* fix rebase issue
* Job count expected in a core config
Besides defaulted 4 * core_count number of workers sourced in the
"tree", the configuration file based value expected and prioritised.
* Multiply the configured value by cpu cores count.
* Don't multiply by the core count twice.
* A couple more usages and a question note.
* Don't multiply a number of jobs to a cpu count
Number of jobs set in a config file directly used to spawn workers.
* Remove the obsolete note.
* Disallow 0 jobs set in a config
* `jobs` config-based value is optional
* Apply suggestions from code review
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* CPU count based default
* Update dvc/tree/base.py
* Happy path unit test
* Update tests/unit/remote/test_remote.py
* Fix the accidentally removed blank line
* Update dvc/tree/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Currently we are converting dir_info to/from lists all the time.
The reason is that dir_info is stored as list of dicts in *.dir files,
but that makes it hard to work with. In addition to that, we will likely
be changing .dir file format in the near future #829, so we need to
abstract away dir_info into something that we won't care how it will be
stored on disk.
Related #3256
Related #4847
* http: added check for HTTP status code 404
Fixes#4288
* http: add unit test and minor change
In tests_exists, changed to raise a HTTPError from dvc instead.
Fixes#4288
* http: removed fixtures in test for HTTPTree.exists
Fixes#4288
* test: add comments on why `raw` is required on the mock
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
* Escape file paths for calculation of md5 hashes on ssh remote
* Add tests for hardlink and improved md5 test with PropertyMock
* Add additional special characters to check for fix of issue #4628 and fix formatting
* Fix linting errors
* Move tests to correct file and use injected ssh_connection instead of setting it up from scratch
When executing `dvc import-url remote://some-webdav/data`, the name of
the WebDAV dependency in the created stage for data used to be `None`.
This resulted in the following error:
```
Traceback (most recent call last):
File "/home/schubert/projects/dvc/dvc/main.py", line 76, in main
ret = cmd.run()
File "/home/schubert/projects/dvc/dvc/command/imp_url.py", line 14, in run
self.repo.imp_url(
File "/home/schubert/projects/dvc/dvc/repo/__init__.py", line 51, in wrapper
return f(repo, *args, **kwargs)
File "/home/schubert/projects/dvc/dvc/repo/scm_context.py", line 4, in run
result = method(repo, *args, **kw)
File "/home/schubert/projects/dvc/dvc/repo/imp_url.py", line 54, in imp_url
stage.run()
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/site-packages/funcy/decorators.py", line 39, in wrapper
return deco(call, *dargs, **dkwargs)
File "/home/schubert/projects/dvc/dvc/stage/decorators.py", line 36, in rwlocked
return call()
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/site-packages/funcy/decorators.py", line 60, in __call__
return self._func(*self._args, **self._kwargs)
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 450, in run
self.save()
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 395, in save
self.md5 = self.compute_md5()
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 388, in compute_md5
m = compute_md5(self)
File "/home/schubert/projects/dvc/dvc/stage/utils.py", line 167, in compute_md5
return dict_md5(
File "/home/schubert/projects/dvc/dvc/utils/__init__.py", line 118, in dict_md5
return dict_hash(d, "md5", **kwargs)
File "/home/schubert/projects/dvc/dvc/utils/__init__.py", line 113, in dict_hash
byts = json.dumps(filtered, sort_keys=True).encode("utf-8")
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
TypeError: '<' not supported between instances of 'str' and 'NoneType'
```
* state: add simple reentrance
* state: pass Repo instead of LocalCache
* state: use granularly
Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
* dvctree: use string paths for performance reasons in
_get_granular_checksum()
* dvctree: cache most recently accessed metadata
* dvctree: cache output dir cache during walk() calls
* diff: use get_file_hash directly instead of get_hash when appropriate
* dvctree: cache most recen dir_cache hashes in _get_granular_checksum
* revert metadata cache changes
- mostly unnecessary after granular checksum changes, will be better
addressed by the future exception handling updates
* fix windows path issue
* HashInfo: store HashInfo.dir_info as a dict instead of list
* DvcTree: use hash_info.dir_info for getting granular file hashes
* use str relpath
* use correct dir_info hash type
* fix windows paths
* Do not require account-level permissions for Azure tree
* Only allow the existence check to pass without error if it failed due to authorization issues
* Update dvc/tree/azure.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/tree/azure.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Restyled by isort
* Update azure.py
Co-authored-by: Matt Hollingsworth <[email protected]>
Co-authored-by: Matt Hollingsworth <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
* tree: validate resolved remote:// sections
* dvc: use checksum_jobs instead of hash_jobs
This is a regression introduced during one of the refactorings.
* allow for a config option for http method
* add tests
* fix tests with method
* Update dvc/tree/http.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
We used to use that logic in DvcTree back before we had proper trees.
Now we have a RepoTree, that combines git/workspace tree with the
DvcTree and dirty state clearly belongs to the git/workspace and not
the dvc itself, as dvc state only depends on dvc-files and lock-files.
Prerequisite for #4523
* diff: use RepoTree to get granular results for dir outs
* update tests for dir file granularity
* diff: prefer repo_tree.get_hash when possible
* DvcTree: handle extra files in dirty dvc out dirs
* support subrepos
This adds support for get/list/api/import/ls on a subrepo.
Also fixes#3180, by adding a granular url support for `dvc.api.get_url`
And, of course it fixes#3369.
* Add tests
* Fix tests, started to get lock_error
* Fix tests for windows
* try to fix mac builds
* Fix comments, readd removed comments
* Fix test
* address review suggestions:
* use `spy` instead of `wraps`
* remove dvc fixture in a test
* set `cache_types` on Git repo
By itself `self.info` is quite confusing, as it is not clear what it is
about. Using `hash_info` makes much more sense and is required to
support alternative hash types.
Related to #4144, #3069, #1676
* dvc: use cache state when saving downloaded cache
Related to #4485
* tree: dont compute hash from scratch for output dirs
Fixes#4485
* tree: fix get_hash
* Speed up dvc files load
See `._load()` implementation. Alternative is not using 3 io calls, but
simply use open and catch exceptions.
Reading is slow for old git revisions.
* tree: git: cache tree and objects
* external: ignore PermissionError on Windows
This is the same thing we have in tests.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev
Saugat Pachhai
Arun MahapatraandRuslan Kuprieiev
Tomás Farías SantanaandRuslan Kuprieiev
Peter Rowlands (변기호)
Ilya PekelnyandRuslan Kuprieiev
LucaButeraandLuca Butera
Liew XunandSaugat Pachhai
simonlousky
Tomás Farías Santana
Samuel Neugber
Frederik Schubert
Guillaume COTER
Robert F. DickersonandRuslan Kuprieiev
Victor
Alexander SchepanovskiandRuslan Kuprieiev