Hi team,
I just [noticed](https://github.com/iterative/dvc/pull/2548#issuecomment-536255251) that there are could be quite a number of false-positives in `dvc/utils/compat.py` because of, well, the things we have to do for compatibility. I would recommend adding that file to `exclude_patterns` to prevent the noise. I've proposed that in this PR.
We've seen some other projects report this as well, and we're working on proactively suggesting this via the product UI to make it easier.
Hope this helps!
Best,
Sanket
Founder, DeepSource
* s3: list_objects -> list_objects_api
* s3: use path_info for _list_paths
* s3: separate _list_paths from _list_objects
* s3: support max items for listing objects/paths
* s3: take into account directories on `exists`
* s3: overwrite `isdir` method
* s3: custom collect directory entry for s3
* s3: strip checksum when collecting directories
* remote: change walk implementation to walk_files
* s3: add comment about isdir logic
* s3: use `dir_path` correctly on `isdir`
* tests: add a few unit tests for S3
* s3: use path_info join instead of posixpath
* tests: use parametrize for s3 tests
* remote: use posixpath accordingly
* tests: remove undeeded AWS variables from S3 test
* tests: rearrange parametrize values for readability
* tests: add encoding, use a single s3 instance, for loop instead of parameterize
* py2: fix encoding
We are incorrectly using `simple_cycle` here, `find_cycle` is much more
suitable for this application and much faster too. For example:
```
$ cat test_networkx.py
import networkx as nx
from dvc.repo.graph import check_acyclic
G = nx.scale_free_graph(1200)
list(nx.find_cycle(G, orientation='ignore'))
list(nx.simple_cycles(G))
$ python -mcProfile -scumtime test_networkx.py &> log.log
```
shows ~100s for `simple_cycles` and ~0s for `find_cycle`.
Fixes#2666
* imp: dedup and parallelize external fetches
This prevents same fetches for external artifacts, groups them by source
and fetches in parallel.
* imp: test that fetch works on non workspace deps
* dvc: fix branch suffixes in status/push/pull
* gc: add --all-commits flag
* gc: fix gc warning
Say that we always leave things used in the working tree
This way we are able to properly utilize automatic connection pools and
not create new fds for each request, which overflows ulimit for max fds
very quickly on mac and windows. Kudos @pared for investigating 🎉Fixes#2600
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This is basically Lock.__init__ copy-paste, except that
instead of using `socket.getfqdn()` we use `socket.gethostname()`
to speed this up. We've seen [1] `getfqdn()` take ~5sec to return
anything, which is way too slow. `gethostname()` is actually a
fallback for `getfqdn()` when it is not able to resolve a
canonical hostname through network. The claimfile that uses
`self._hostname` is still usable, as it uses `pid` and random
number to generate the resulting lock file name, which is unique
enough for our application.
[1] https://github.com/iterative/dvc/issues/2582Fixes#2582
Make `Repo.checkout` accept a list of targets, instead of having to feed
it targets one-by-one. This allows us to properly handle checkout
errors.
Fixes#2580
* dvc: dedup used cache
This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts
* dvc: delay checksum names concatenation
* get/import/url: handle existing output directory
Currently if you run
```
dvc get https://github.com/iterative/example-get-started model.pkl --out .
```
it will nuke your cwd. To fix that we need to properly handle existing
directories, same as usual unix cli utilities do that.
* travis: specify python version when installing from chocolatey
Fixes#2609
Imagine the scenario of shared cache. To set it up, user would do:
```
dvc config cache.dir /path
dvc config cache.shared group
```
Now, when he runs any dvc operation that involves the Repo instance,
RemoteLOCAL will also be initialized and it will currently auto-create
the directory with a regular `os.mkdir`, which is not using group
permissions. To solve that, we can't simply use `self.makdirs` instead,
as it will cause same issues if you run any dvc command before
`dvc config cache.shared group`. So what we should do is create
directories on demand, instead of creating it in advance, same as we do
in other remotes.
The change itself is pretty trivial and involved removing that `mkdir`
line, adjusting tests to not rely on cache dir existence and also
adjusting `dvc version` to not rely on cache directory existance.
Fixes#2563
Sanket Saurav
Mr. Outis
Ruslan Kuprieiev
Casper da Costa-Luis
Dashamir Hoxha
Alexander Schepanovski
Srividya Krishnakumar
Ruslan Kuprieiev
Nabanita Dash
Alok Kumar Sahoo
SOFIA SUNAM
Kirill Malev
Aman Sharma