Commit Graph
4 Commits
Author SHA1 Message Date
Peter Rowlands (변기호) 74bd8137ba remote: finish separating RemoteTree methods (#3946)
* remote: move upload()/download() into tree

* remote: move path_info/path_cls into tree

* remote.azure: finish moving methods into tree

* remote.gs: finish moving methods into tree

* remote.hdfs: finish moving methods to tree

* remote.http: finish moving methods into tree

* remote.oss: finish moving methods into tree

* remote.s3: finish moving methods into tree

* remote.ssh: finish moving methods into tree

* remote.gdrive: finish moving methods into tree

* tests: update remote unit tests

* tests: update func tests

* remote: use walk_files() for all remotes

* list_cache_paths() now uses tree.walk_files() for all remotes except
  local/ssh

* fix DS warnings

* bugfixes
2020-06-04 23:22:31 +03:00
Peter Rowlands (변기호) 833e82a7ea Drop Python 3.5 support (#3792)
* drop python 3.5 support

* dvc: remove fspath() use inside os.path calls

* dvc: replace dvc.compat.fspath with os.fspath

* tests: replace dvc.compat.fspath with os.fspath

* utils: relpath() should use fspath/str

* utils: os.readlink doesn't support path-like obj on Windows until 3.8
2020-05-14 01:46:42 +03:00
nik123 fefbcc0b4c refactor: dvc/remotes name unification (#3684)
Partially fixes #2089
2020-04-26 20:55:00 +03:00
Peter Rowlands (변기호) b04964f4b7 remote: locally index list of checksums available on cloud remotes (#3634)
* remote: locally index list of checksums available on cloud remotes

* fix review issues

- use pytest fixtures
- use sha256 digest
- s/fd/fobj/
- check more specific IO errors

* remote: use .dir checksums to validate index

* command: add --drop-index option for push/pull/status

* tests: --drop-index unit tests

* remote: make index threadsafe

* remote: force re-index after gc

* only save/dump index once per command

* remote: add helper index functions update_all/replace_all

* remote: store index as sqlite3 database

* fix deepsource warnings

* rename --drop-index to --clear-index

* rename index.invalidate to index.clear

* functional tests for remote.index

* fix index not being updated on status -c

* remote: add index.intersection()

* remote: use index when making assumptions about remote .dir contents

* add missing func tests

* do not index standalone files

* cleanup index test fixture

* fix deepsource warnings

* update autocompletion scripts/cli help message for --clear-index

* tests: remove unused mocker

* push: only index successfully pushed dirs

- do not index files on partial push/upload

* remote: revert behavior to trust .dir files on remote

- skip unnecessary index check for dir contents if .dir file exists on
  the remote

* fix missing RemoteIndexNoop functions

* separate index validation from status dir exists query

* bugfix: include indexed checksums for all cache_exists cases

* review fix: cleanup debug messages

* tests: fix DS warning

* remove --clear-index

* remove unused index functions
2020-04-21 15:13:45 +03:00