Default chunk size for gs is 100M, which might be too much for
particular network. So if we are getting ConnectionError, we
should try lowering the chunk size until we reach the minimum
allowed chunk size of 256K. Also note that `chunk_size` must be a
multiple of 256K per the API specification.
Fixes#2572
* s3: list_objects -> list_objects_api
* s3: use path_info for _list_paths
* s3: separate _list_paths from _list_objects
* s3: support max items for listing objects/paths
* s3: take into account directories on `exists`
* s3: overwrite `isdir` method
* s3: custom collect directory entry for s3
* s3: strip checksum when collecting directories
* remote: change walk implementation to walk_files
* s3: add comment about isdir logic
* s3: use `dir_path` correctly on `isdir`
* tests: add a few unit tests for S3
* s3: use path_info join instead of posixpath
* tests: use parametrize for s3 tests
* remote: use posixpath accordingly
* tests: remove undeeded AWS variables from S3 test
* tests: rearrange parametrize values for readability
* tests: add encoding, use a single s3 instance, for loop instead of parameterize
* py2: fix encoding
* dvc: dedup used cache
This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts
* dvc: delay checksum names concatenation
Unlike PyInstaller, pyenv doesn't leave backups of original env vars
when it modifies them. If we look into the shim, pyenv and pyenv-exec,
we can figure out that the PATH is modified like this:
PATH=$PYENV_BIN_PATH:${bin_path}:${plugin_bin}:$PATH
where
PYENV_BIN_PATH - starts with $PYENV_ROOT, see pyenv-exec source code.
bin_path - might not start with $PYENV_ROOT as it runs realpath on
it, see pyenv source code.
plugin_bin - might contain more than 1 entry, which start with
$PYENV_ROOT, see pyenv source code.
So having this, we can make a rightful assumption about what parts of the PATH
we need to remove in order to get the original PATH.
Fixes#2506
This is an extension of our callback stages (no deps, so always
considered as changed) for stages with dependencies, so that these
stages could be used properly in the middle of the DAG. See attached
issue for more info.
Related to #2378
* http: add exception when trying to traverse remote
Fix#2380
* http: exception raise on `list_cache_paths` instead of `all`
`NotImplementedError` -> `RemoteActionNotImplemented`
* http: raise exception on `__init__`
* set default no_traverse to True on remote base
* tests: explicitly use traverse on data cloud and no traverse with HTTP
* remote: define a new RemoteConfigError
* remote: reuse ConfigError instead of RemoteConfigError
* tests: set no_traverse to False with SSHMocked
* deepsource: fix minor issue with spacing
* remove blankspace
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Current test is testing for signal handler restoration in an obscure
way. As a result, the test wasn't actually testing anything, because
KeyboardInterrupt is raised by default_int_handler, and so raising it
in a mock won't be caught by signal handlers, because they are dealing
with real singnals.
Fix#2406
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev
GeorgeSabu-MEC
Aman Sharma
Mr. Outis
Jorge Orpinel
Casper da Costa-Luis
Alexander Schepanovski
Ruslan Kuprieiev
pawel
joshua postel