Commit Graph
11 Commits
Author SHA1 Message Date
Saugat Pachhai b9000ed926 pre-commit: update black to 21.5b1 (#6032)
* pre-commit: update black to 21.5b1

also updated others, but black is more noticeable

* remove magic trailing comma
2021-05-18 15:51:15 +00:00
Batuhan Taskaya d4a3df73df s3: fsspec migration (#5683)
* s3: migrate to s3fs

* s3: strip etag

* s3: disable instance cache

* tests: use the proper remote:// form in test_checkout

* tests: move/drop some tests

* setup: depend on s3fs

* tests: normalize s3 tests with some cleanups

* Use _download/_upload with old implementation

* fsspec: cache .exists() results

* setup: pin the boto version

* setup: install s3fs[boto3]

* s3: proper exception management

* s3: config improvements

* setup: reference without quotes

* setup: install boto separately

* setup: install moto server

* s3: mocking adjustments for moto_server

* s3: mark some fixtures with scope=session

* s3: s3_fake_creds_file scopes to func

* fsspec: add some docs regarding exists() optimization

* fsspec: exists() fallback

* fsspec: revert exists() optimization

* tests: ssl_verify => verify

* setup: temporarily fetch from master

* s3: add boto as a requirement

* tests: do an extra check regarding race conditions

* tests: use motoserver docker image

* tests: wrap config deletion with try/finally

* tests: disable s3 by default

* setup: upgrade fsspec
2021-05-05 11:36:10 +03:00
Batuhan Taskaya 11b0581536 import-url: support importing empty directories on object storage services (#5536)
* import-url: support importing empty directories on object storage services

Empty directories are basically files with a trailing slash at the end of their names,
with empty contents on object service providers (such as azure, google storage). This
patch implements the `is_empty()` method for the `FSSpecWrapper` to determining whether
a path is empty (directory or a file) and also tweaks `.isdir()` in order to see empty
directories.

The base class is only used by azure and gcsfs for now, but if we go on the route of
migrating normal providers (like proper filesystems, not just object storages), we might
need to do these tweaks on a subclass (like `ObjectStorageFSSpecWrapper` etc), but I
don't think it is neccesary as of now.

The `is_empty` method is already there, and also used in the importing part to
warn users that they are importing something empty (https://github.com/iterative/dvc/blob/e385e59a098cd661ea1ea4257c9a5735c5c32bc6/dvc/output/base.py#L252-L254)
so this is not a new method. Just for the reference, as far as I can locate there
is no variant of this method in the fsspec.

* tests: revert custom mkdir for cloud storages

* tests: allign assingments

* add some comments
2021-03-12 18:11:08 +02:00
Saugat Pachhai 21200c14df refactor webdav to use client; extract write_text to base (#5411) 2021-02-04 16:48:25 +05:45
Batuhan Taskaya ce554b6ae7 tree: Introduce the new .ls() API (#5275)
* s3: faster directory hashing

* Introduce Tree.info() API

* tests for s3

* Implement support for azure

* New .ls() API

* remove _iter_hashes from other trees

* assert recursive

* yield strings instead of file_infos

* Add GSTree.ls()

* Add DETAIL_FIELDS to the GSTree

* WebDAVTree.ls()

* proper webdav root directory initalization

* disable arguments-differ
2021-02-04 12:10:42 +02:00
Saugat Pachhai 5faae2a6e9 test: allow selectively running tests for remotes (#4920)
Some of the tests are run by default, other requires
`--enable-{remote}` to run. The remotes that are run by default
also have a `--disable-{remote}` flag. All of the remotes can be
enabled using `--all` flag.

To use this functionality in the fixture,
`test_config` fixture has to be used, and requires to call
`test_config.requires(remote)`.
In an individual test, you can simply use `pytest.mark.remote`
to make them skip or run.
2020-11-23 11:43:39 +05:45
Paweł Redzyński d3931def53 tests: real_s3 fixture: use specific path (#4884) 2020-11-12 16:36:08 +01:00
Ruslan Kuprieiev 30212f886b tests: create remote fixture (#4163)
Usually we need it to parametrize tests with, so there is no
reason to duplicate the code. I've left `local_remote` fixture
just because it is so convenient and often used in a lot of tests.
2020-07-03 22:30:56 +03:00
Ruslan Kuprieiev 07da9f1e5c tests: remotes: use TmpDir-like fixtures (#4140) 2020-07-02 04:46:40 +03:00
Saugat Pachhai ad306d7bf3 Enable most of the pylint checks (#4111) 2020-06-25 18:10:14 +03:00
Ruslan Kuprieiev 51a8c782ae tests: create remote fixtures (#4063)
* tests: use remote fixtures

* tests: use remote fixtures for data cloud tests

* tests: treat remotes as tmp dirs
2020-06-18 01:36:13 +03:00