Casper da Costa-Luis and Peter Rowlands
0578643f08
machine: move to tpi[ssh] ( #6528 )
...
* machine: move to tpi[ssh]
- depends on https://github.com/iterative/tpi
* lazy import tpi
* fix lint warning
Co-authored-by: Peter Rowlands <[email protected]
2021-09-03 14:37:24 +09:00
Peter Rowlands (변기호)
9fc4c60749
machine: add dvc machine ssh ( #6501 )
...
* machine: add naive asyncssh shell client
* machine: add `dvc machine ssh` command
* add ssh requirements to terraform install extras
* update asyncssh comments
* fix py <3.7
* fix async mocking in py<3.8
* use mock instead of unittest.mock
2021-09-01 16:20:13 +09:00
Saugat Pachhai
ea2a38a207
restrict typing_extensions version for py3.10 ( #6503 )
2021-08-30 17:19:39 +05:45
Saugat Pachhai
5b9bdac965
add terraform to tests dependencies ( #6502 )
2021-08-30 17:04:47 +05:45
Peter Rowlands (변기호)
4ee98299ee
dvc machine: implement initial terraform support (#6462 )
...
* tpi: add naive terraform-provider-iterative wrapper
* executor: add naive backend/terraform wrapper
* setup: add dvc[terraform] install extras
* config: add feature.executor feature flag
* fix named executor config sections
* executor: add poc add/modify/remove/init/destroy commands
* relax python-terraform requirement
* ci: install terraform extras
* tpi: support .tf.json format
* s/executor/machine
* terraform: use separate dirs per machine
* s/init/create/
* reserve local machine name
* add initial ssh skeleton
* add initial machine unit tests
* add initial machine command tests
* add tests for add/remove
* enable feature by default in test env
* close temp pemfile object so it can be used on windows
* cleanup pemfile syntax
2021-08-27 12:46:24 +03:00
Ruslan Kuprieiev
7da20a7c74
setup: upgrade s3fs
...
Fixes https://github.com/iterative/dvc/issues/6466
2021-08-20 19:43:14 +03:00
Paweł Redzyński
a7175501aa
ruamel.yaml: adjust to new DuplicatedKeyError ( #6464 )
2021-08-19 20:02:16 +03:00
Batuhan Taskaya
8406e605ab
fs: find(..., detail=True) now casts values properly ( #6432 )
...
* fs: find(..., detail=True) now casts values properly
* bump pydrive2
2021-08-16 14:26:07 +03:00
Saugat Pachhai
1bb7f39e14
add support for Python 3.10 ( #6422 )
...
* add support for Python 3.10
pyarrow is not available, so we are skipping tests for it.
And, restricting `hdfs` extras to be not installed in Python3.10.
Pathlib removed `_init` method and `init` parameter, so just working
around that. It's not a big change.
* skip windows on 3.10; update pip/wheel/setuptools during install
pywin32 is not yet available for 3.10 Python, so skipping those for now
* install libgit2 with Python3.10 as pygit2 wheel is not available
* install libgit2
* only run on macOS, libgit2-1.1 not available on ubuntu
2021-08-16 16:24:28 +05:45
Saugat Pachhai
a7f01b65a5
Unpin networkx
...
Networkx v2.6.2 now does not require any additional requirements. All of the other 2.6 releases have been yanked. So, it's safe for us to unpin it.
2021-08-13 14:53:10 +05:45
Batuhan Taskaya
1f8278252a
gdrive: migrate to pydrive2.fs ( #6399 )
2021-08-10 12:59:44 +03:00
Saugat Pachhai
7444696dcd
Fix merging of params when empty ( #6378 )
...
* Fix merging of params when empty
Recent release of python-benedict==0.24.1 started ignoring empty dictionaries
during merge. The way we expected to work was questionable and also how
benedict itself tries to handle the references to empty dictionary is
also questionable.
I consider this to be a proper way to handle this even if this gets fixed
on the benedict side.
* Revert "setup: limit python-benedict version (#6377 )"
This reverts commit 15de826266
2021-08-02 22:45:04 +03:00
Ruslan Kuprieiev
15de826266
setup: limit python-benedict version ( #6377 )
...
Temporary fix for #6374 Kudos @skshetry
2021-08-02 20:44:03 +03:00
Batuhan Taskaya
9634efeb84
ssh: handle concurrency issues for raw command execution ( #6370 )
...
* ssh: handle concurrency issues for raw command execution
* setup: bump sshfs
2021-07-29 14:40:49 +03:00
Ruslan Kuprieiev
e3a541aa00
setup: relax aiobotocore requirements ( #6367 )
...
Fixes #6366
2021-07-28 16:19:16 +03:00
Batuhan Taskaya
9b4b6e5b31
ssh: migrate to sshfs ( #6295 )
2021-07-28 12:49:53 +03:00
Paweł Redzyński
67b42ebe4f
plots: cleanup data extraction ( #6355 )
...
* plots: cleanup data extraction
* fixup
2021-07-23 10:28:36 +09:00
Batuhan Taskaya
1d7e754044
setup: bump flatten-dict to 0.4.1 ( #6357 )
2021-07-22 21:46:18 +03:00
Saugat Pachhai
5faf9b3a7d
update webdav4 to latest fsspec version ( #6343 )
2021-07-21 10:17:14 +00:00
Batuhan Taskaya
66d0df2d12
setup: upgrade adlfs to 2021.7.1 ( #6337 )
...
The previous release got yanked on the PyPI, due to it containing backwards incompatible changes.
2021-07-20 08:59:45 +00:00
Christian Clauss
d8b67f6f78
Upgrade to modern Python syntax ( #6296 )
2021-07-19 01:30:37 +03:00
Ruslan Kuprieiev
97809661bd
setup: pathspec <0.9.0 ( #6331 )
2021-07-18 18:24:28 +03:00
Batuhan Taskaya
27c1722b90
setup: bump ossfs==2021.7.5 ( #6326 )
2021-07-16 20:48:57 +08:00
Batuhan Taskaya
6c8673b158
setup: upgrade fsspec/s3fs/gcsfs to 2021.7.0 ( #6317 )
2021-07-15 11:33:07 +02:00
Gao and Ruslan Kuprieiev
93f36d9351
oss: migrate to ossfs ( #6307 )
...
* Migrate to ossfs
fix 6177
* Add verification to object stored on oss
* Change the oss remote's dependency.
pass all of the tests
* Performance optimaztion and ossfs version update
* Remove the argument to set the resumable get/put
1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default
* Use a new api for performance optimization
* Without filesize progress bar version
* With progress bar version
* Bump and add comments
1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger
* Change the name of kwrags following parent class.
* Bump to new ossfs and rewrite upload_fobj
Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj
* Optimization on rm_files
* Some optimization follow the old version.
* Update dvc/fs/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected] [email protected] [email protected]
2021-07-13 15:27:34 +03:00
Batuhan Taskaya
58f0911234
azure: support directory imports with azure ( #6245 )
...
* azure: support directory imports with azure
* upgrade azurite to 3.13
* tmp: use azurite 3.12?
* upgrade to azurite 3.13.1
* bump adlfs to 2021.7.0
2021-07-13 13:39:30 +03:00
Ruslan Kuprieiev
bd5809572e
setup: relax requirements for importlib-metadata
2021-07-08 23:01:48 +03:00
Batuhan Taskaya
f90b9cdee1
doctor: show versions for fs requirements ( #6273 )
...
* doctor: show versions for fs requirements
* plugins => supports
2021-07-07 21:07:16 +03:00
Jon Crall
e72354e98e
Loosen versions for fsspec and pygtrie ( #6242 )
2021-06-28 23:57:49 +03:00
Ruslan Kuprieiev
4e792ae61c
setup: update fsspec to 2021.6.1 ( #6219 )
...
* setup: update fsspec to 2021.6.1
* fs: local: remove temp workaround
2021-06-24 10:06:42 +05:45
Saugat Pachhai and Ruslan Kuprieiev
6e5e62857b
Bump fsspec version ( #6214 )
...
* Bump fsspec version
* local: use fsspec 2021.06.0 with a temp hack
Co-authored-by: Ruslan Kuprieiev <[email protected]
2021-06-23 16:07:32 +03:00
Saugat Pachhai
ce3a3ab100
pass size to upload_fobj ( #6187 )
...
This allows us to select between chunked and non-chunked uploading
in Webdav, and hopefully in the http itself.
2021-06-22 13:16:15 +03:00
Saugat Pachhai
df67dc2db8
webdav: bring back support for user in url ( #6155 )
...
Also added some optimizations, and merged Webdavs into the same file.
2021-06-11 19:52:42 +05:45
Batuhan Taskaya
0deddf81cf
setup: bump fsspec/s3fs/gcsfs to 2021.06.0 ( #6137 )
...
* setup: bump fsspec/s3fs/gcsfs to 2021.06.0
* bump fsspec to upstream
2021-06-10 15:17:43 +03:00
Saugat Pachhai
ed2a0b4282
webdav4: use webdav4 ( #6099 )
2021-06-04 16:42:26 +03:00
Saugat Pachhai
bf5deb08e4
pin networkx to 2.5 ( #6085 )
...
I am bit hesitant to add an upper-bound here
as networkx seems to be requiring a lot of
dependencies from next version onwards.
2021-05-31 21:39:48 +03:00
Batuhan Taskaya
50ec64b841
setup: bump fsspec/s3fs/gcsfs to 2021.5.0 ( #6056 )
2021-05-25 14:54:56 +03:00
Ruslan Kuprieiev
aa672c5108
setup: bump dulwich to 0.20.23 ( #6053 )
...
Fixes #6052
2021-05-25 00:30:22 +03:00
Peter Rowlands (변기호)
9dda6bd32c
git: update dulwich to 0.20.22 ( #6049 )
...
* git: update dulwich to 0.20.22
* git: disable reading system config in CI
2021-05-24 17:40:14 +09:00
Batuhan Taskaya
e7571ab325
setup: fixate aiobotocore/botocore/boto3 versions ( #5987 )
2021-05-10 18:28:11 +03:00
Batuhan Taskaya
d4a3df73df
s3: fsspec migration ( #5683 )
...
* s3: migrate to s3fs
* s3: strip etag
* s3: disable instance cache
* tests: use the proper remote:// form in test_checkout
* tests: move/drop some tests
* setup: depend on s3fs
* tests: normalize s3 tests with some cleanups
* Use _download/_upload with old implementation
* fsspec: cache .exists() results
* setup: pin the boto version
* setup: install s3fs[boto3]
* s3: proper exception management
* s3: config improvements
* setup: reference without quotes
* setup: install boto separately
* setup: install moto server
* s3: mocking adjustments for moto_server
* s3: mark some fixtures with scope=session
* s3: s3_fake_creds_file scopes to func
* fsspec: add some docs regarding exists() optimization
* fsspec: exists() fallback
* fsspec: revert exists() optimization
* tests: ssl_verify => verify
* setup: temporarily fetch from master
* s3: add boto as a requirement
* tests: do an extra check regarding race conditions
* tests: use motoserver docker image
* tests: wrap config deletion with try/finally
* tests: disable s3 by default
* setup: upgrade fsspec
2021-05-05 11:36:10 +03:00
Batuhan Taskaya
353e4cf374
fsspec: update to 0.9 (including remotes) ( #5788 )
2021-04-09 14:12:25 +03:00
Batuhan Taskaya
ddb1ece283
setup: fix all fsspec remote versions ( #5769 )
2021-04-06 10:54:13 +03:00
Saugat Pachhai
7b25a9de24
Update pycryptodome to 3.10 ( #5762 )
...
3.10 has wheels for Python3.9 in Windows.
2021-04-03 12:10:24 +05:45
Batuhan Taskaya
e78375e3d9
setup: bump adlfs to 0.7 ( #5760 )
2021-04-02 13:35:15 +03:00
Alex Lopez
4e7f924e8b
dvc: fix breakage with rich==10.0.0 ( #5733 )
...
* dvc: fix breakage with rich==10.0.0
Change in the arguments of __rich_measure__.
Fixes #5722
* dvc: Fix linting breakage from rich==10.0.0
Fixes : #5733
* dvc: set lower bound for rich dependency to 10.0.0
Fixes #5722
2021-03-30 20:57:40 +03:00
Peter Rowlands (변기호)
262816c20e
git: update dulwich to 0.20.21 ( #5730 )
2021-03-30 21:23:14 +09:00
Saugat Pachhai
f6b04c54e0
deps: limit rich version to <10 ( #5723 )
...
The recent release of rich 10.0.0 broke some internal APIs
that we were depending on for exp show.
See https://github.com/iterative/dvc/issues/5722
2021-03-29 12:09:02 +05:45
Saugat Pachhai
1931a6ec1e
Move tests requirements to test_requirements.txt ( #5674 )
...
Plan is to add dependabot to update those packages required for testing
2021-03-26 14:13:43 +02:00
Saugat Pachhai
dc3fe6fbec
Update pylint-pytest to 1.0.3 ( #5673 )
2021-03-23 15:28:33 +05:45
Casper da Costa-LuisandPeter Rowlands
Saugat Pachhai
Ruslan Kuprieiev
Paweł Redzyński
Batuhan Taskaya
Batuhan Taskaya
Christian Clauss
GaoandRuslan Kuprieiev
Jon Crall
Alex Lopez