Default chunk size for gs is 100M, which might be too much for
particular network. So if we are getting ConnectionError, we
should try lowering the chunk size until we reach the minimum
allowed chunk size of 256K. Also note that `chunk_size` must be a
multiple of 256K per the API specification.
Fixes#2572
* s3: list_objects -> list_objects_api
* s3: use path_info for _list_paths
* s3: separate _list_paths from _list_objects
* s3: support max items for listing objects/paths
* s3: take into account directories on `exists`
* s3: overwrite `isdir` method
* s3: custom collect directory entry for s3
* s3: strip checksum when collecting directories
* remote: change walk implementation to walk_files
* s3: add comment about isdir logic
* s3: use `dir_path` correctly on `isdir`
* tests: add a few unit tests for S3
* s3: use path_info join instead of posixpath
* tests: use parametrize for s3 tests
* remote: use posixpath accordingly
* tests: remove undeeded AWS variables from S3 test
* tests: rearrange parametrize values for readability
* tests: add encoding, use a single s3 instance, for loop instead of parameterize
* py2: fix encoding
* imp: dedup and parallelize external fetches
This prevents same fetches for external artifacts, groups them by source
and fetches in parallel.
* imp: test that fetch works on non workspace deps
* dvc: fix branch suffixes in status/push/pull
* gc: add --all-commits flag
* gc: fix gc warning
Say that we always leave things used in the working tree
Make `Repo.checkout` accept a list of targets, instead of having to feed
it targets one-by-one. This allows us to properly handle checkout
errors.
Fixes#2580
* dvc: dedup used cache
This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts
* dvc: delay checksum names concatenation
* get/import/url: handle existing output directory
Currently if you run
```
dvc get https://github.com/iterative/example-get-started model.pkl --out .
```
it will nuke your cwd. To fix that we need to properly handle existing
directories, same as usual unix cli utilities do that.
* travis: specify python version when installing from chocolatey
Fixes#2609
Imagine the scenario of shared cache. To set it up, user would do:
```
dvc config cache.dir /path
dvc config cache.shared group
```
Now, when he runs any dvc operation that involves the Repo instance,
RemoteLOCAL will also be initialized and it will currently auto-create
the directory with a regular `os.mkdir`, which is not using group
permissions. To solve that, we can't simply use `self.makdirs` instead,
as it will cause same issues if you run any dvc command before
`dvc config cache.shared group`. So what we should do is create
directories on demand, instead of creating it in advance, same as we do
in other remotes.
The change itself is pretty trivial and involved removing that `mkdir`
line, adjusting tests to not rely on cache dir existence and also
adjusting `dvc version` to not rely on cache directory existance.
Fixes#2563
* ui/perf: do not clone same repo multiple times
Closes#2490
External repos are still created in /tmp. They might be reused until
main() end. If no perfect match is found then a repo with the same url
is copied (not cloned again) and adjusted to fit a request: rev and
cache_dir.
NOTE: test_download_error_pulling_imported_stage() was modified to not
rely on clone error.
* test: fix import update test
* test: fix external_repo test under Windows
* dvc: fix non-master default branch for external repos
* dvc: remove unused .unset_dir() method
* tree: make open method use mode and encoding
* api: try to open using self.tree if output not found
* repo: rename OutputMissingException
* test: api: test adding not cached output
* api: open: exists test instead of try
* repo: open: try opening from tree first
* repo: open: handle missing and not cached files
* repo: open: separate find out logic from opening
* checkout: raise on non existing dir cache
* checkout: don't throw on failed load_dir_cache
* checkout: raise on failed checkouts, only after all checkouts are perfromed
* remote: base: raise on failed dir cache loading
* rename exception for failed checkout
Unlike PyInstaller, pyenv doesn't leave backups of original env vars
when it modifies them. If we look into the shim, pyenv and pyenv-exec,
we can figure out that the PATH is modified like this:
PATH=$PYENV_BIN_PATH:${bin_path}:${plugin_bin}:$PATH
where
PYENV_BIN_PATH - starts with $PYENV_ROOT, see pyenv-exec source code.
bin_path - might not start with $PYENV_ROOT as it runs realpath on
it, see pyenv source code.
plugin_bin - might contain more than 1 entry, which start with
$PYENV_ROOT, see pyenv source code.
So having this, we can make a rightful assumption about what parts of the PATH
we need to remove in order to get the original PATH.
Fixes#2506
* wip - pull is working
* use fetch instead of reproduce
* test pull and fetch
* style changes
* fetch: catch config error on fetch instead of get_remote
* output: remove unused guard clause
* fetch: define a more specific error instead of ConfigError
* fetch: display number of errors correctly
* exceptions: remove "Failed" prefix to download/upload
* fetch: catch other exceptions that might halt the download
* fetch: log catched exceptions
* fetch: reformat exception
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
* fix formatting
Leftover code was causing problems when working with symlinks, because
it tried to use `realpath` on symlinks to further use it to determine
parent directory.
Fixes#2476
This is an extension of our callback stages (no deps, so always
considered as changed) for stages with dependencies, so that these
stages could be used properly in the middle of the DAG. See attached
issue for more info.
Related to #2378
Ivan ShchekleinandRuslan Kuprieiev
Jorge Orpinel
Aman Sharma
Mr. Outis
Casper da Costa-Luis
Alexander Schepanovski
Ruslan Kuprieiev
Paweł Redzyński