Commit Graph
364 Commits
Author SHA1 Message Date
Paweł Redzyński 67b42ebe4f plots: cleanup data extraction (#6355)
* plots: cleanup data extraction

* fixup
2021-07-23 10:28:36 +09:00
Batuhan Taskaya 1d7e754044 setup: bump flatten-dict to 0.4.1 (#6357) 2021-07-22 21:46:18 +03:00
Saugat Pachhai 5faf9b3a7d update webdav4 to latest fsspec version (#6343) 2021-07-21 10:17:14 +00:00
Batuhan Taskaya 66d0df2d12 setup: upgrade adlfs to 2021.7.1 (#6337)
The previous release got yanked on the PyPI, due to it containing backwards incompatible changes.
2021-07-20 08:59:45 +00:00
Christian Clauss d8b67f6f78 Upgrade to modern Python syntax (#6296) 2021-07-19 01:30:37 +03:00
Ruslan Kuprieiev 97809661bd setup: pathspec <0.9.0 (#6331) 2021-07-18 18:24:28 +03:00
Batuhan Taskaya 27c1722b90 setup: bump ossfs==2021.7.5 (#6326) 2021-07-16 20:48:57 +08:00
Batuhan Taskaya 6c8673b158 setup: upgrade fsspec/s3fs/gcsfs to 2021.7.0 (#6317) 2021-07-15 11:33:07 +02:00
GaoandRuslan Kuprieiev 93f36d9351 oss: migrate to ossfs (#6307)
* Migrate to ossfs

fix 6177

* Add verification to object stored on oss

* Change the oss remote's dependency.

pass all of the tests

* Performance optimaztion and ossfs version update

* Remove the argument to set the resumable get/put

1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default

* Use a new api for performance optimization

* Without filesize progress bar version

* With progress bar version

* Bump and add comments

1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger

* Change the name of kwrags following parent class.

* Bump to new ossfs and rewrite upload_fobj

Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj

* Optimization on rm_files

* Some optimization follow the old version.

* Update dvc/fs/oss.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/objects/db/oss.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* bump into ossfs 2021.7.2

* Bump into ossfs 2021.7.3OC

* Switch to real_oss_server

* Switch to a real oss server

* Use the oss bucket located in US.

* Remove some emulator related code

* Add envs to ci

* remove some oss pytest XFAIL

Because we didn't use emulator, we can remove some pytest XFAILs

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-07-13 15:27:34 +03:00
Batuhan Taskaya 58f0911234 azure: support directory imports with azure (#6245)
* azure: support directory imports with azure

* upgrade azurite to 3.13

* tmp: use azurite 3.12?

* upgrade to azurite 3.13.1

* bump adlfs to 2021.7.0
2021-07-13 13:39:30 +03:00
Ruslan Kuprieiev bd5809572e setup: relax requirements for importlib-metadata 2021-07-08 23:01:48 +03:00
Batuhan Taskaya f90b9cdee1 doctor: show versions for fs requirements (#6273)
* doctor: show versions for fs requirements

* plugins => supports
2021-07-07 21:07:16 +03:00
Jon Crall e72354e98e Loosen versions for fsspec and pygtrie (#6242) 2021-06-28 23:57:49 +03:00
Ruslan Kuprieiev 4e792ae61c setup: update fsspec to 2021.6.1 (#6219)
* setup: update fsspec to 2021.6.1

* fs: local: remove temp workaround
2021-06-24 10:06:42 +05:45
Saugat PachhaiandRuslan Kuprieiev 6e5e62857b Bump fsspec version (#6214)
* Bump fsspec version

* local: use fsspec 2021.06.0 with a temp hack

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-06-23 16:07:32 +03:00
Saugat Pachhai ce3a3ab100 pass size to upload_fobj (#6187)
This allows us to select between chunked and non-chunked uploading
in Webdav, and hopefully in the http itself.
2021-06-22 13:16:15 +03:00
Saugat Pachhai df67dc2db8 webdav: bring back support for user in url (#6155)
Also added some optimizations, and merged Webdavs into the same file.
2021-06-11 19:52:42 +05:45
Batuhan Taskaya 0deddf81cf setup: bump fsspec/s3fs/gcsfs to 2021.06.0 (#6137)
* setup: bump fsspec/s3fs/gcsfs to 2021.06.0

* bump fsspec to upstream
2021-06-10 15:17:43 +03:00
Saugat Pachhai ed2a0b4282 webdav4: use webdav4 (#6099) 2021-06-04 16:42:26 +03:00
Saugat Pachhai bf5deb08e4 pin networkx to 2.5 (#6085)
I am bit hesitant to add an upper-bound here
as networkx seems to be requiring a lot of
dependencies from next version onwards.
2021-05-31 21:39:48 +03:00
Batuhan Taskaya 50ec64b841 setup: bump fsspec/s3fs/gcsfs to 2021.5.0 (#6056) 2021-05-25 14:54:56 +03:00
Ruslan Kuprieiev aa672c5108 setup: bump dulwich to 0.20.23 (#6053)
Fixes #6052
2021-05-25 00:30:22 +03:00
Peter Rowlands (변기호) 9dda6bd32c git: update dulwich to 0.20.22 (#6049)
* git: update dulwich to 0.20.22

* git: disable reading system config in CI
2021-05-24 17:40:14 +09:00
Batuhan Taskaya e7571ab325 setup: fixate aiobotocore/botocore/boto3 versions (#5987) 2021-05-10 18:28:11 +03:00
Batuhan Taskaya d4a3df73df s3: fsspec migration (#5683)
* s3: migrate to s3fs

* s3: strip etag

* s3: disable instance cache

* tests: use the proper remote:// form in test_checkout

* tests: move/drop some tests

* setup: depend on s3fs

* tests: normalize s3 tests with some cleanups

* Use _download/_upload with old implementation

* fsspec: cache .exists() results

* setup: pin the boto version

* setup: install s3fs[boto3]

* s3: proper exception management

* s3: config improvements

* setup: reference without quotes

* setup: install boto separately

* setup: install moto server

* s3: mocking adjustments for moto_server

* s3: mark some fixtures with scope=session

* s3: s3_fake_creds_file scopes to func

* fsspec: add some docs regarding exists() optimization

* fsspec: exists() fallback

* fsspec: revert exists() optimization

* tests: ssl_verify => verify

* setup: temporarily fetch from master

* s3: add boto as a requirement

* tests: do an extra check regarding race conditions

* tests: use motoserver docker image

* tests: wrap config deletion with try/finally

* tests: disable s3 by default

* setup: upgrade fsspec
2021-05-05 11:36:10 +03:00
Batuhan Taskaya 353e4cf374 fsspec: update to 0.9 (including remotes) (#5788) 2021-04-09 14:12:25 +03:00
Batuhan Taskaya ddb1ece283 setup: fix all fsspec remote versions (#5769) 2021-04-06 10:54:13 +03:00
Saugat Pachhai 7b25a9de24 Update pycryptodome to 3.10 (#5762)
3.10 has wheels for Python3.9 in Windows.
2021-04-03 12:10:24 +05:45
Batuhan Taskaya e78375e3d9 setup: bump adlfs to 0.7 (#5760) 2021-04-02 13:35:15 +03:00
Alex Lopez 4e7f924e8b dvc: fix breakage with rich==10.0.0 (#5733)
* dvc: fix breakage with rich==10.0.0

Change in the arguments of __rich_measure__.

Fixes #5722

* dvc: Fix linting breakage from rich==10.0.0

Fixes: #5733

* dvc: set lower bound for rich dependency to 10.0.0

Fixes #5722
2021-03-30 20:57:40 +03:00
Peter Rowlands (변기호) 262816c20e git: update dulwich to 0.20.21 (#5730) 2021-03-30 21:23:14 +09:00
Saugat Pachhai f6b04c54e0 deps: limit rich version to <10 (#5723)
The recent release of rich 10.0.0 broke some internal APIs
that we were depending on for exp show.
See https://github.com/iterative/dvc/issues/5722
2021-03-29 12:09:02 +05:45
Saugat Pachhai 1931a6ec1e Move tests requirements to test_requirements.txt (#5674)
Plan is to add dependabot to update those packages required for testing
2021-03-26 14:13:43 +02:00
Saugat Pachhai dc3fe6fbec Update pylint-pytest to 1.0.3 (#5673) 2021-03-23 15:28:33 +05:45
Saugat Pachhai 0b810fb256 Advertise Python 3.9 support in classifiers 2021-03-16 13:37:37 +05:45
nik123 6ac7def6f1 setup: remove duplicate 'pytest-cov' dependency (#5617)
* setup: remove duplicate 'pytest-cov' dependency

* setup: returned pytest-cov version
2021-03-14 14:08:11 +05:45
Saugat Pachhai e709437cd2 update pylint and pin dependencies (#5603)
Started having some issues with pylint-pytest, pinning it as well
2021-03-11 15:24:27 +05:45
Peter Rowlands (변기호) d68f6d3696 pygit/dulwich: implement stash functionality (#5286)
* pygit2: disable set_ref with reflog message

* git: implement is_dirty in dulwich

* pygit2: implement _stash_apply

* pygit: update to 1.5.0

* enable set_ref(message=...) for pygit2

* git: replace backend reflog_delete in favor of _stash_drop

* pygit: support stash_push with custom refs

* update tests for backend stash_drop behavior

* pygit: raise MergeConflictError on failed stash apply

* pygit: reload index from disk before stash apply

* update stash_apply checkout strategy

* mark flaky test
2021-03-08 17:24:35 +02:00
Batuhan Taskaya f7d8761734 gdrive: use the normal file backend for saved credentials (#5554)
* gdrive: sync with pydrive's save backend changes

* bump pydrive version

* better temporary credentials managements for service accounts

* proper temp var management
2021-03-08 16:30:05 +02:00
af37527560 setup: rename azure.indentity to azure-identity (#5548)
* simple like that

* Update dvc/version.py

Co-authored-by: Marc-Antoine Belanger <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-03-05 01:43:32 +02:00
Peter Rowlands (변기호) 1e7df0c27e dulwich: update to 0.20.20 (#5545) 2021-03-04 12:37:19 +02:00
Ruslan Kuprieiev 803a100f32 setup: move google-cloud-storage to test requirements (#5540) 2021-03-03 17:11:30 +02:00
Batuhan Taskaya 2982e98c81 gs: Use rm() instead of delete() calls (#5535)
delete is a direct alias to rm(), though apparently it is kind of
broken on the latest release of gcsfs. This doesn't change any behavior
but only uses rm() directly instead of the given alias.
2021-03-02 13:53:44 +02:00
Batuhan Taskaya b65dfe04f9 gs: Migrate Google Storage to FSSpec (#5526)
* azure: generalize fsspec functionality into FSSpecWrapper

* gs: migrate google storage to fsspec

* tests: disable empty_dir tests

* setup: add gcsfs to requirements

* gs: decode md5Hash into md5

* gs: speed up downloads via increasing chunk size
2021-03-01 17:15:31 +02:00
Ruslan Kuprieiev 502f0eecc9 tests: temporarily skip dvclive tests (#5513) 2021-02-24 08:52:33 +02:00
Ruslan Kuprieiev 8f568ab197 state: use diskcache (#5505)
* state: only load when needed

* state: don't use separate load/dump

* checkout: group link removal logic together

* state: use diskcache.Cache

* staging: use pbar and save to state

* state: don't pass the whole repo to __init__

* staging: explicitly pass state object

* fs: get rid of odb property

* state: don't close the connection to cache

* repo: close: close state
2021-02-23 20:51:34 +02:00
Batuhan Taskaya 174621696d azure: migrate AzureFileSystem to fsspec (#5463)
* azure: migrate AzureFileSystem to fsspec

* client => fs

* principal auth

* better login error handling
2021-02-22 12:05:15 +02:00
Paweł Redzyński 63f32936b2 live integration: turn on tests (#5487) 2021-02-19 09:25:28 +01:00
Batuhan Taskaya 9ae8bcdf52 gdrive: support service auth with json (#5485)
* gdrive: support service auth with json

* retry logic
2021-02-18 17:10:35 +02:00
Ruslan Kuprieiev 282e83c481 tests: temporarily exclude gdrivefs from extras (#5483)
When trying to publish to pypi, we get:

```
Invalid value for requires_dist. Error: Can't have direct dependency:
"gdrivefs @ git+https://github.com/intake/gdrivefs.git ; extra ==
'tests'"
```
2021-02-17 05:49:34 +02:00