All three of those use more or less the same logic for testing different
link types and maybe then using them. So now instead of having 3
implementations, we'll have just one.
Filesystem `transfer` now will not try to catch particular errno's, but
instead will just ignore any OSError and move down to the next link type
to try out, and if the last one fails - it will raise the last exception,
so if it is a common problem - we'll be able to see it.
This patch also removes `odb.confirmed_cache_type`, as it was caching
link type that is not really a cache-level thing, but rather a checkout thing,
that should only be cached for particular objects we are linking, as multiple
objects could be linked to different filesystems from the same odb (e.g.
external local output on hdd and local data on your ssd).
Next step here will be to introduce metadata diff, so that we can get rid of
temporary Link wrapper and introduce Transfer and corresponding transfer
config, that will know what links to use and could have some other things
like callbacks, so we don't continue passing dozens of kwargs along the stack.
Continuing from #6963Fixes#6980
Intermediate PR in removing path_info from our code base.
It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.
For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).
Fixes#6774
* rebased PR 6111 onto master
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use make_tmp_dir fixture in tests
Co-authored-by: U119511 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Alexander Meier <[email protected]>
Co-authored-by: meierale <[email protected]>
* fs.download: add callback support
* fix lint issues
* compat for gdrive, use fs to upload text for tests
* fix typing issues
* skip gdrive test
* Update tests/func/test_fs.py
* fs.upload: add support for optional callback
Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* objects: migrate remote status to objects.status
* data_cloud: use objects.status
* objects: migrate remote._process to objects.transfer
* data_cloud: use objects.transfer
* objects: handle memfs staging objects in transfer
* fetch/import: use unified transfer()
* update get_remote and transfer exception usage
* update tests
* objects.transfer: use src ODB verify rules after xfer
* odb: move index from remote into odb
* objects.transfer: skip src status check when possible
* update tests
* remove dvc.remote
* odb.index: migrate to diskcache
* objects: fix status/transfer optimization
* update index usage
* state: remove dead sqlite related code
* update cloud tests
* drop unnecessary remove in tests
* use abstract base class in odb index
* status: load trees from local cache instead of remote odb when possible
* handle status when dir cache is explicitly removed
* Migrate to ossfs
fix 6177
* Add verification to object stored on oss
* Change the oss remote's dependency.
pass all of the tests
* Performance optimaztion and ossfs version update
* Remove the argument to set the resumable get/put
1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default
* Use a new api for performance optimization
* Without filesize progress bar version
* With progress bar version
* Bump and add comments
1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger
* Change the name of kwrags following parent class.
* Bump to new ossfs and rewrite upload_fobj
Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj
* Optimization on rm_files
* Some optimization follow the old version.
* Update dvc/fs/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/objects/db/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* bump into ossfs 2021.7.2
* Bump into ossfs 2021.7.3OC
* Switch to real_oss_server
* Switch to a real oss server
* Use the oss bucket located in US.
* Remove some emulator related code
* Add envs to ci
* remove some oss pytest XFAIL
Because we didn't use emulator, we can remove some pytest XFAILs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* azure: use a custom event loop for authentication
* add test_temp_event_loop
* wrap open() too!
* ignore aiohttp warnings for normal level use-cases
* rename tests/unit/remote/test_azure to tests/unit/fs/test_azure
we just need to mark them with needs_internet marker.
Unfortunately, it's not possible to do that automatically
with the fixtures as fixtures are per-session and we need
something on more higher level (which the marker is not passed
through in the pytest).
* tree: Implement upload_fobj protocol to azure
* tree: Implement upload_fobj protocol to google storage
* tree: Implement upload_fobj protocol to google drive
* tree: Implement upload_fobj protocol to hdfs
* tree: Implement upload_fobj protocol to oss (alibaba cloud)
* tree: Implement upload_fobj protocol to webhdfs (also some cleanup)
* tree: Implement upload_fobj protocol to http/https
* Add _generate_download_url() to the HTTP (for tests)
* tree: Implement a temporary upload_fobj protocol to webdav/webdavs
* Unify progress bars
* Lazily evaluate the file size, for http uploads
* Remove copy_fobj protocol and _transfer_file_as_whole
* show progress bar on cache transfer
* Simplify s3 upload
* Remove CHUNK_SIZE (except for some special cases)
* remove concurrency restrictions
* Update dvc/tree/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Add support for WebHDFS
* Style and linting fixes
* Implement abstract methods and disable no-member pylint errors
* Add init unit test
* Yield client and fix redefinition of hdfs
* Get dynamic hdfs Docker port
* Add support for open in WebHDFSTree
* Remove hdfscli_config fallbacks in WebHDFSTree
* Use port 50070 for webhdfs
* WebHDFSTree supports http URL and proper walking
* Fix failing unit test after refactoring
* Add overwrite argument to upload
* Add remaining func tests for webhdfs
* Remove print statements
* Allow dynamic attributes in pylint for hdfs library
* Comment error on instantiating HDFS client from config
* Remove depth argument from call to walk
* Update dvc/tree/webhdfs.py
* Remove unnecessary deque
* Check exception messages hdfs client failure from config
* Return client directly
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Split long line
* Add size to HashInfo
* Implement move via hdfs.rename
* Add missing line
* Make parent directories before moving file in WebHDFS
* Add total to progress bar
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Job count expected in a core config
Besides defaulted 4 * core_count number of workers sourced in the
"tree", the configuration file based value expected and prioritised.
* Multiply the configured value by cpu cores count.
* Don't multiply by the core count twice.
* A couple more usages and a question note.
* Don't multiply a number of jobs to a cpu count
Number of jobs set in a config file directly used to spawn workers.
* Remove the obsolete note.
* Disallow 0 jobs set in a config
* `jobs` config-based value is optional
* Apply suggestions from code review
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* CPU count based default
* Update dvc/tree/base.py
* Happy path unit test
* Update tests/unit/remote/test_remote.py
* Fix the accidentally removed blank line
* Update dvc/tree/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* http: added check for HTTP status code 404
Fixes#4288
* http: add unit test and minor change
In tests_exists, changed to raise a HTTPError from dvc instead.
Fixes#4288
* http: removed fixtures in test for HTTPTree.exists
Fixes#4288
* test: add comments on why `raw` is required on the mock
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
* Escape file paths for calculation of md5 hashes on ssh remote
* Add tests for hardlink and improved md5 test with PropertyMock
* Add additional special characters to check for fix of issue #4628 and fix formatting
* Fix linting errors
* Move tests to correct file and use injected ssh_connection instead of setting it up from scratch
Ruslan Kuprieiev
Batuhan Taskaya
Guðmundur HeimissonandRuslan Kuprieiev
Nikita Kodenko
Saugat Pachhai
Batuhan Taskaya
Peter Rowlands (변기호)
Christian Clauss
GaoandRuslan Kuprieiev
Paweł Redzyński
Dmitry Kropachev
Tomás Farías SantanaandRuslan Kuprieiev
Ilya PekelnyandRuslan Kuprieiev
Liew XunandSaugat Pachhai
Samuel Neugber
Guillaume COTER