* remote: move upload()/download() into tree
* remote: move path_info/path_cls into tree
* remote.azure: finish moving methods into tree
* remote.gs: finish moving methods into tree
* remote.hdfs: finish moving methods to tree
* remote.http: finish moving methods into tree
* remote.oss: finish moving methods into tree
* remote.s3: finish moving methods into tree
* remote.ssh: finish moving methods into tree
* remote.gdrive: finish moving methods into tree
* tests: update remote unit tests
* tests: update func tests
* remote: use walk_files() for all remotes
* list_cache_paths() now uses tree.walk_files() for all remotes except
local/ssh
* fix DS warnings
* bugfixes
* remote: remove unpacked dir
We've done a lot of optimizations lately, which made unpacked dir trick
obsoleted.
* local: save us a stat call in is_protected
* local: speedup checksum_to_path ~x5.5
* tree: walk: don't use os.path.join
It is ~5.5 times slower than joining by hand.
* dvcignore: optimize matching x10
On a repo with a dvcignore with 1 pattern and a directory with 400K
files, `dvc status` now takes ~8 sec instead of ~30 sec.
To achieve that, we make some assumptions about the paths formats that
we are dealing with, so we could use simpler logic instead of using
very slow `relpath`, `abspath` etc on every entry in a directory.
It is also clear that CleanTree behavior is inconsistent (even tests
expect very different outputs from it), so we will need to look into
this later.
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
* remote: locally index list of checksums available on cloud remotes
* fix review issues
- use pytest fixtures
- use sha256 digest
- s/fd/fobj/
- check more specific IO errors
* remote: use .dir checksums to validate index
* command: add --drop-index option for push/pull/status
* tests: --drop-index unit tests
* remote: make index threadsafe
* remote: force re-index after gc
* only save/dump index once per command
* remote: add helper index functions update_all/replace_all
* remote: store index as sqlite3 database
* fix deepsource warnings
* rename --drop-index to --clear-index
* rename index.invalidate to index.clear
* functional tests for remote.index
* fix index not being updated on status -c
* remote: add index.intersection()
* remote: use index when making assumptions about remote .dir contents
* add missing func tests
* do not index standalone files
* cleanup index test fixture
* fix deepsource warnings
* update autocompletion scripts/cli help message for --clear-index
* tests: remove unused mocker
* push: only index successfully pushed dirs
- do not index files on partial push/upload
* remote: revert behavior to trust .dir files on remote
- skip unnecessary index check for dir contents if .dir file exists on
the remote
* fix missing RemoteIndexNoop functions
* separate index validation from status dir exists query
* bugfix: include indexed checksums for all cache_exists cases
* review fix: cleanup debug messages
* tests: fix DS warning
* remove --clear-index
* remove unused index functions
Replaced Path* dataclasses with PathInfo/URLInfo/CloudURLInfo, which
know how parse, serialize, stringify and join themselves. They also
provide useful accessors like part of an url or pathlib.Path like
.parent or .name.
Along the way two families of bugs where fixed and should never occur in
the future:
- paths not matched as strings anymore, e.g. using str.startswith()
- Output.path is eliminated as ambiguous
Both families might had caused undefined surprising behavior when local
and remote paths clash.
Peter Rowlands (변기호)
Ruslan Kuprieiev
Saugat Pachhai
Ivan ShchekleinandJorge Orpinel
nik123
Mr. Outis
pawel
Alexander Schepanovski