* azure: use a custom event loop for authentication
* add test_temp_event_loop
* wrap open() too!
* ignore aiohttp warnings for normal level use-cases
* rename tests/unit/remote/test_azure to tests/unit/fs/test_azure
* objects: merge transfer() with stage()
* objects: drop streamfile
* objects: preserve the signature of get_file_hash
* objects: override hash_jobs if jobs= passed
* objects: support no progress bar directive
* tests: spy on _get_file_hash
* objects: update=... directive
* update: only transfer missing caches with --to-remote
* tests: remove local_remote from the primary test (due to it moves)
* objects: add move=True/False option
* objects: implement _get_file_obj on top of get_file_hash
* objects: use threading on save()
* objects: pass jobs= on to-cache as well
* objects: use in place tqdm
* objects: check for FileExistsError
* Update dvc/objects/__init__.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* import-url: support importing empty directories on object storage services
Empty directories are basically files with a trailing slash at the end of their names,
with empty contents on object service providers (such as azure, google storage). This
patch implements the `is_empty()` method for the `FSSpecWrapper` to determining whether
a path is empty (directory or a file) and also tweaks `.isdir()` in order to see empty
directories.
The base class is only used by azure and gcsfs for now, but if we go on the route of
migrating normal providers (like proper filesystems, not just object storages), we might
need to do these tweaks on a subclass (like `ObjectStorageFSSpecWrapper` etc), but I
don't think it is neccesary as of now.
The `is_empty` method is already there, and also used in the importing part to
warn users that they are importing something empty (https://github.com/iterative/dvc/blob/e385e59a098cd661ea1ea4257c9a5735c5c32bc6/dvc/output/base.py#L252-L254)
so this is not a new method. Just for the reference, as far as I can locate there
is no variant of this method in the fsspec.
* tests: revert custom mkdir for cloud storages
* tests: allign assingments
* add some comments
* gdrive: sync with pydrive's save backend changes
* bump pydrive version
* better temporary credentials managements for service accounts
* proper temp var management
delete is a direct alias to rm(), though apparently it is kind of
broken on the latest release of gcsfs. This doesn't change any behavior
but only uses rm() directly instead of the given alias.
Switch from `md5` to `etag` as the checksum parameter. Due to `etag`
is not preserved while doing server side copies, this patch drops support
for external gs outputs.
* s3: faster directory hashing
* Introduce Tree.info() API
* tests for s3
* Implement support for azure
* New .ls() API
* remove _iter_hashes from other trees
* assert recursive
* yield strings instead of file_infos
* Add GSTree.ls()
* Add DETAIL_FIELDS to the GSTree
* WebDAVTree.ls()
* proper webdav root directory initalization
* disable arguments-differ
* add: implement saving straight to cache
* Adjust CLI error cases to handle -o
* make tests pass on windows
* support explicit --jobs
* simplify the implementation a bit
* polish the tests
* tree: Implement upload_fobj protocol to azure
* tree: Implement upload_fobj protocol to google storage
* tree: Implement upload_fobj protocol to google drive
* tree: Implement upload_fobj protocol to hdfs
* tree: Implement upload_fobj protocol to oss (alibaba cloud)
* tree: Implement upload_fobj protocol to webhdfs (also some cleanup)
* tree: Implement upload_fobj protocol to http/https
* Add _generate_download_url() to the HTTP (for tests)
* tree: Implement a temporary upload_fobj protocol to webdav/webdavs
* Unify progress bars
* Lazily evaluate the file size, for http uploads
* Remove copy_fobj protocol and _transfer_file_as_whole
* show progress bar on cache transfer
* Simplify s3 upload
* Remove CHUNK_SIZE (except for some special cases)
* remove concurrency restrictions
* Update dvc/tree/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Implement export-to-remote functionality
* Redistrubition of functionality from tree to remote
* Pass chunk size into the open_url
* Move functionality away from remote to cache
* Basic progress bars for both transfer types
* Proper job calculation
* Redesign .open() usage in order to work with different trees
* Align the signatures of other open() functions to accept **kwargs
* Implement chunked uploading to the ssh
* Use local cache to compute directory hash (md5)
* S3: multipart upload test
* Wrap local cache's hash_to_path with PathInfo
* Use .isdir() instead of a try/except statement for calculating the from_infos
* New CLI layout (add/import-url)
* Adjust CLI tests to accept new arguments
* tests: import_url/add func/unit tests
* Introduce cache.transfer()
* Invalid CLI combination tests
* Store dir_info key in the transfer_directory
* LocalTree.upload_fobj
* Tests for utils
* Lazy imports on some places
* --to-remote migration (and repo._transfer)
* Use proper temporary location, more efficient pbar calc
* Directory transfer logic rewrite
* Use shutil.copyfileobj()
* Rebase...
* Better tests
* Inline transfer operation to .add() / .transfer()
* Open in the 'rb' mode for the tests
* More tests, better invalid option handling at repo level
* .read() certain amount of bytes to validate the content
* Better invalid argument handling
* Better progress bars by using .getsize() API
* For fallback upload, use the from_info.name instead of temporary naem
* introduce --jobs for add/import-url
* S3Tree._list_paths now recognizes the max_items limit
* Use modern approach + add a test
* refactor tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* lint
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Fix partial file checkouts in the same directory
* unittest => pytest
* Try to rationalize changed() to take filter_info
* Use path.read_text() to verify the directory structure
* Try to use specialized hash, fall back to general one if doesn't exist
* Cover subdirectories in hash specialization
* coverage: add a test for empty subdirectories
* Move out the dir_info filtering block
* hash_origin => path_info
* cleanup
* Move hashing functionality after the existence checks
* Simplify it a bit
* Update dvc/cache/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions from code review
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions
* simplify tests
* change filenames
* Some visual changes for consistency
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Batuhan Taskaya
