* master:
address @efiop's suggestions
s3: add support for top level empty directories
remote: http: calculate length basing on response and not head call
test: remote: http: clear test file name
HTTPError: reformat error message
Update dvc/remote/http.py
remote: http: rename http error, refactor test
remote: http: raise exception when download response with error status code
Support GDrive as remote
setup: don't forget contextlib2
remote: refactor ssh ask password code
remote: protect all remote client/session creation code with locks
test: refactor & remove redundant test fixtures
NoRemoteInExternalRepoError: dont pass cause of exception
perf: optimize cache listing for local, ssh and hdfs
remote: small .save_info()/.get_checksum() cleanup
Default chunk size for gs is 100M, which might be too much for
particular network. So if we are getting ConnectionError, we
should try lowering the chunk size until we reach the minimum
allowed chunk size of 256K. Also note that `chunk_size` must be a
multiple of 256K per the API specification.
Fixes#2572
* s3: list_objects -> list_objects_api
* s3: use path_info for _list_paths
* s3: separate _list_paths from _list_objects
* s3: support max items for listing objects/paths
* s3: take into account directories on `exists`
* s3: overwrite `isdir` method
* s3: custom collect directory entry for s3
* s3: strip checksum when collecting directories
* remote: change walk implementation to walk_files
* s3: add comment about isdir logic
* s3: use `dir_path` correctly on `isdir`
* tests: add a few unit tests for S3
* s3: use path_info join instead of posixpath
* tests: use parametrize for s3 tests
* remote: use posixpath accordingly
* tests: remove undeeded AWS variables from S3 test
* tests: rearrange parametrize values for readability
* tests: add encoding, use a single s3 instance, for loop instead of parameterize
* py2: fix encoding
* dvc: dedup used cache
This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts
* dvc: delay checksum names concatenation
* http: add exception when trying to traverse remote
Fix#2380
* http: exception raise on `list_cache_paths` instead of `all`
`NotImplementedError` -> `RemoteActionNotImplemented`
* http: raise exception on `__init__`
* set default no_traverse to True on remote base
* tests: explicitly use traverse on data cloud and no traverse with HTTP
* remote: define a new RemoteConfigError
* remote: reuse ConfigError instead of RemoteConfigError
* tests: set no_traverse to False with SSHMocked
* deepsource: fix minor issue with spacing
* remove blankspace
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Related to #1629
Speeds up HDFS tests significantly. E.g. ExternalHDFS test goes from 600
sec to 190sec. test_open_external[HDFS] from 160sec to 3sec.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev
Vera Sativa
VeraandRuslan Kuprieiev
Alexander Schepanovski
Saugat Pachhai
Mr. Outis
pawel
Max Risuhin
Casper da Costa-Luis
Ruslan Kuprieiev
GeorgeSabu-MEC
joshua postel