* s3,gs: don't append "/" to prefixed path when listing objects
* remote: use optional prefix kwarg for walk_files
* if prefix is True, path will be walked as a prefix
* if prefix is False, path will be treated as a directory (trailing
slash will be appended for S3-like remotes)
* tests: add unit test for BaseRemoteTree.list_paths
* remote: move constants into trees
* remote: replace Remote() Cache() helper functions
* output: update for remote changes
* cache/data_cloud: update for remote changes
* dependency: update for remote changes
* stage: update for remote changes
* remote: move remaining non-index methods into tree
* tests: update remote unit tests
* remote: LocalCache needs to lock remote index, not cache index
* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH
* tests: repo tree open needs state context
* remote: make Remote.gc a class method
* operates on obj.tree, so that GC can be done with both remote and
cache instances
* bug fixes
* tests: update test_data_cloud
* bug fixes
* tests: update func tests
* remote: move get_file_checksum() into tree
* makes RemoteTree and RepoTree consistent with regard to checksum
calculation
* remote: save() now takes explicit tree parameter
* when tree is remote.tree, save will be a move + link operation
(same as default existing behavior)
* when saving path from a different tree, save will be a copy operation
* tests: update for moved remote/tree functions
* remote: move get_checksum into tree
* remote: separate cloud remote and cache classes
* tests: update unit tests for remote/cache separation
* remote: cloud cache should extend cloud remote
* remote: move LocalRemote.get
* tests: update func tests
* dependency: update for moved get_checksum
* remote: fix state lookup bug
* remote: move upload()/download() into tree
* remote: move path_info/path_cls into tree
* remote.azure: finish moving methods into tree
* remote.gs: finish moving methods into tree
* remote.hdfs: finish moving methods to tree
* remote.http: finish moving methods into tree
* remote.oss: finish moving methods into tree
* remote.s3: finish moving methods into tree
* remote.ssh: finish moving methods into tree
* remote.gdrive: finish moving methods into tree
* tests: update remote unit tests
* tests: update func tests
* remote: use walk_files() for all remotes
* list_cache_paths() now uses tree.walk_files() for all remotes except
local/ssh
* fix DS warnings
* bugfixes
* BaseRemote: start moving tree functions into BaseRemoteTree
* remote: add LocalRemoteTree
* remote: add cloud remote trees
* remote: move remove(), makedirs() into tree
* remote: move move(), copy(), symlink(), hardlink(), reflink() into tree
* fix attributes for moved remote methods
* tests: update unit tests for moved remote tree methods
* tests: mv unit/remote/test_remote_dir.py unit/remote/test_remote_tree.py
* fix attributes for moved remote tree functions
* tests: update func tests for moved remote tree methods
* test fixes
* fix DS warnings
* remote: locally index list of checksums available on cloud remotes
* fix review issues
- use pytest fixtures
- use sha256 digest
- s/fd/fobj/
- check more specific IO errors
* remote: use .dir checksums to validate index
* command: add --drop-index option for push/pull/status
* tests: --drop-index unit tests
* remote: make index threadsafe
* remote: force re-index after gc
* only save/dump index once per command
* remote: add helper index functions update_all/replace_all
* remote: store index as sqlite3 database
* fix deepsource warnings
* rename --drop-index to --clear-index
* rename index.invalidate to index.clear
* functional tests for remote.index
* fix index not being updated on status -c
* remote: add index.intersection()
* remote: use index when making assumptions about remote .dir contents
* add missing func tests
* do not index standalone files
* cleanup index test fixture
* fix deepsource warnings
* update autocompletion scripts/cli help message for --clear-index
* tests: remove unused mocker
* push: only index successfully pushed dirs
- do not index files on partial push/upload
* remote: revert behavior to trust .dir files on remote
- skip unnecessary index check for dir contents if .dir file exists on
the remote
* fix missing RemoteIndexNoop functions
* separate index validation from status dir exists query
* bugfix: include indexed checksums for all cache_exists cases
* review fix: cleanup debug messages
* tests: fix DS warning
* remove --clear-index
* remove unused index functions
Windows (or rather NTFS) doesn't allow deleting read-only files, which made us
try to chmod +w first. But that is not necessery on Linux and Mac, and
has led to unnecessary cache file mode changes, which was causing issues
in shared cache scenario when a user that doesn't own a file was trying
to automatically protect it.
Fixes#3618
* remote: use progress bar when paginating
- add `progress_callback` parameter to `list_cache_paths()` so that
remotes can update remote cache traverse progress bar after fetching a
page
* Show remote size estimation status as no-total progress bar
* Update pbar per item rather than per page
* make tests more explicit (don't use mock.ANY)
* Apply suggestions from code review
minor fixes/tidy
* Use progress_callback in ssh/local list_cache_paths()
- keep ssh/local consistent with other remotes (even though `dvc
gc`/`RemoteBASE.all()` do not currently use progress bars)
* Fix review issues
Co-authored-by: Casper da Costa-Luis <[email protected]>
* s3/gs: do not try to make directory on top level
* tests: unittest makedirs for s3 and gs
* remove setattr and use patch.object
* Update tests/unit/remote/test_remote.py
* remote: Optimize traverse/no_traverse behavior
- estimate remote file count by fetching a single parent cache dir, then
determine whether or not to use no_traverse method for checking
remainder of cache entries in `cache_exists`
- thread requests when traversing full remote file lists (by fetching
one parent cache dir per thread)
* Use arbitrary prefix length for traverse remote cache queries
- default to 3
- for remotes that only support per-directory query use 2 (gdrive, hdfs)
* Add unit tests for new traverse/no_traverse behavior
* Fix deepsource warning
* Fix review issues
* fix CI
* Fix review issues
* obsolete no_traverse config option
- remove RemoteBASE.DEFAULT_NO_TRAVERSE and replace it with
RemoteBASE.CAN_TRAVERSE (True for all remotes except http)
* review issues - use 'object exists' instead of 'no_traverse'
* fix obsolete no_traverse test issues
* fix SSHMocked func tests
* remote.http: support _upload() via HTTP POST
- uploaded files are sent as chunked encoding POST data
Fixes#3247
* remote.http: add support for HTTP basic and digest auth
- username, password, ask_password options work the same way as for
ssh remotes
- basic_auth (bool) and digest_auth (bool) options added for http(s)
remotes
- digest_auth takes precedence over basic_auth if both are enabled
* remote.http: auth_method() unit tests
* tests: Add functional tests for HTTP remotes
- HTTP remotes now tested locally using a SimpleHTTPServer instance that
allows reading/writing to a temp directory
* tests: fix http tests on py3.5, 3.6
* remote.http: add custom auth method
* s/custom_header/custom_auth_header/
* Apply suggestions from code review
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
Ruslan Kuprieiev
Peter Rowlands (변기호)
Saugat Pachhai
Jack WellsandRuslan Kuprieiev
Casper da Costa-Luis
Ivan ShchekleinandJorge Orpinel
nik123
Max Risuhin
Alexander Schepanovski
Guro Bokum
pawel
Aman Sharma
Mr. Outis