339 Commits
Author SHA1 Message Date
Batuhan Taskaya 353e4cf374 fsspec: update to 0.9 (including remotes) (#5788) 2021-04-09 14:12:25 +03:00
Batuhan Taskaya ddb1ece283 setup: fix all fsspec remote versions (#5769) 2021-04-06 10:54:13 +03:00
Saugat Pachhai 7b25a9de24 Update pycryptodome to 3.10 (#5762)
3.10 has wheels for Python3.9 in Windows.
2021-04-03 12:10:24 +05:45
Batuhan Taskaya e78375e3d9 setup: bump adlfs to 0.7 (#5760) 2021-04-02 13:35:15 +03:00
Alex Lopez 4e7f924e8b dvc: fix breakage with rich==10.0.0 (#5733)
* dvc: fix breakage with rich==10.0.0

Change in the arguments of __rich_measure__.

Fixes #5722

* dvc: Fix linting breakage from rich==10.0.0

Fixes: #5733

* dvc: set lower bound for rich dependency to 10.0.0

Fixes #5722
2021-03-30 20:57:40 +03:00
Peter Rowlands (변기호) 262816c20e git: update dulwich to 0.20.21 (#5730) 2021-03-30 21:23:14 +09:00
Saugat Pachhai f6b04c54e0 deps: limit rich version to <10 (#5723)
The recent release of rich 10.0.0 broke some internal APIs
that we were depending on for exp show.
See https://github.com/iterative/dvc/issues/5722
2021-03-29 12:09:02 +05:45
Saugat Pachhai 1931a6ec1e Move tests requirements to test_requirements.txt (#5674)
Plan is to add dependabot to update those packages required for testing
2021-03-26 14:13:43 +02:00
Saugat Pachhai dc3fe6fbec Update pylint-pytest to 1.0.3 (#5673) 2021-03-23 15:28:33 +05:45
Saugat Pachhai 0b810fb256 Advertise Python 3.9 support in classifiers 2021-03-16 13:37:37 +05:45
nik123 6ac7def6f1 setup: remove duplicate 'pytest-cov' dependency (#5617)
* setup: remove duplicate 'pytest-cov' dependency

* setup: returned pytest-cov version
2021-03-14 14:08:11 +05:45
Saugat Pachhai e709437cd2 update pylint and pin dependencies (#5603)
Started having some issues with pylint-pytest, pinning it as well
2021-03-11 15:24:27 +05:45
Peter Rowlands (변기호) d68f6d3696 pygit/dulwich: implement stash functionality (#5286)
* pygit2: disable set_ref with reflog message

* git: implement is_dirty in dulwich

* pygit2: implement _stash_apply

* pygit: update to 1.5.0

* enable set_ref(message=...) for pygit2

* git: replace backend reflog_delete in favor of _stash_drop

* pygit: support stash_push with custom refs

* update tests for backend stash_drop behavior

* pygit: raise MergeConflictError on failed stash apply

* pygit: reload index from disk before stash apply

* update stash_apply checkout strategy

* mark flaky test
2021-03-08 17:24:35 +02:00
Batuhan Taskaya f7d8761734 gdrive: use the normal file backend for saved credentials (#5554)
* gdrive: sync with pydrive's save backend changes

* bump pydrive version

* better temporary credentials managements for service accounts

* proper temp var management
2021-03-08 16:30:05 +02:00
af37527560 setup: rename azure.indentity to azure-identity (#5548)
* simple like that

* Update dvc/version.py

Co-authored-by: Marc-Antoine Belanger <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-03-05 01:43:32 +02:00
Peter Rowlands (변기호) 1e7df0c27e dulwich: update to 0.20.20 (#5545) 2021-03-04 12:37:19 +02:00
Ruslan Kuprieiev 803a100f32 setup: move google-cloud-storage to test requirements (#5540) 2021-03-03 17:11:30 +02:00
Batuhan Taskaya 2982e98c81 gs: Use rm() instead of delete() calls (#5535)
delete is a direct alias to rm(), though apparently it is kind of
broken on the latest release of gcsfs. This doesn't change any behavior
but only uses rm() directly instead of the given alias.
2021-03-02 13:53:44 +02:00
Batuhan Taskaya b65dfe04f9 gs: Migrate Google Storage to FSSpec (#5526)
* azure: generalize fsspec functionality into FSSpecWrapper

* gs: migrate google storage to fsspec

* tests: disable empty_dir tests

* setup: add gcsfs to requirements

* gs: decode md5Hash into md5

* gs: speed up downloads via increasing chunk size
2021-03-01 17:15:31 +02:00
Ruslan Kuprieiev 502f0eecc9 tests: temporarily skip dvclive tests (#5513) 2021-02-24 08:52:33 +02:00
Ruslan Kuprieiev 8f568ab197 state: use diskcache (#5505)
* state: only load when needed

* state: don't use separate load/dump

* checkout: group link removal logic together

* state: use diskcache.Cache

* staging: use pbar and save to state

* state: don't pass the whole repo to __init__

* staging: explicitly pass state object

* fs: get rid of odb property

* state: don't close the connection to cache

* repo: close: close state
2021-02-23 20:51:34 +02:00
Batuhan Taskaya 174621696d azure: migrate AzureFileSystem to fsspec (#5463)
* azure: migrate AzureFileSystem to fsspec

* client => fs

* principal auth

* better login error handling
2021-02-22 12:05:15 +02:00
Paweł Redzyński 63f32936b2 live integration: turn on tests (#5487) 2021-02-19 09:25:28 +01:00
Batuhan Taskaya 9ae8bcdf52 gdrive: support service auth with json (#5485)
* gdrive: support service auth with json

* retry logic
2021-02-18 17:10:35 +02:00
Ruslan Kuprieiev 282e83c481 tests: temporarily exclude gdrivefs from extras (#5483)
When trying to publish to pypi, we get:

```
Invalid value for requires_dist. Error: Can't have direct dependency:
"gdrivefs @ git+https://github.com/intake/gdrivefs.git ; extra ==
'tests'"
```
2021-02-17 05:49:34 +02:00
Corentin Hembise 52ee995f5e Update requirements of dulwich to 0.20.19 (#5480)
This should fix #5438
2021-02-17 10:12:04 +09:00
Ruslan Kuprieiev e90c5fdd19 fs: remove unused use_state decorator (#5469)
* fs: remove unused use_state decorator

* pylint: limit astroid version
2021-02-16 02:00:55 +02:00
Batuhan TaskayaandRuslan Kuprieiev f4b14c2d4a gdrive: Implement .ls() protocol / proper tree walking with .walk_files() (#5433)
* gdrive: .ls()/.walk()/.walk_dir()

* better walking

* proper tests

* ._ls_recursive()

* use posxipath for tests

* store ids instead of raw names

* add gdrive fixture

* add dependency

* tests: temporarily disable gdrive tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-02-13 03:10:38 +02:00
Peter Rowlands (변기호) b370465209 exp show: customize rich table rendering behavior (#5381)
* exp show: use rich native pager support

* exp show: re-enable column header styles

* utils: add custom classes to override rich table collapse behavior

* set final collapsed column width to 1 to allow ellipsis
2021-02-09 23:34:04 +09:00
Ruslan Kuprieiev 226c68f8ff dvc: introduce File/Tree cache objects (#5424) 2021-02-09 04:00:06 +02:00
Peter Rowlands (변기호) 550cbb5e11 git: update dulwich dependency to 0.20.18 (#5420) 2021-02-05 20:43:44 +09:00
Saugat Pachhai b2c2929869 pyarrow support for Python3.9 (#5391) 2021-02-02 17:31:11 +02:00
Ruslan Kuprieiev 6a1f3659f5 tests: limit pytest version by <6.2.2 (#5333) 2021-01-25 22:01:54 +02:00
Ruslan Kuprieiev 9203d33a3c reqs: tests: remove awscli (#5306) 2021-01-21 18:09:35 +02:00
Ruslan Kuprieiev d038bca05e tests: add basic [web]hdfs mockers (#5279) 2021-01-19 20:03:16 +02:00
Asif Saif UddinandRuslan Kuprieiev f8c9797f2a add psutil to install requirement (#5232)
* add  psutil to install requirement

* info: remove unneded checks

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-18 12:30:38 +02:00
Saugat Pachhai 709e121b31 package: Increment shtab's version (#5212) 2021-01-06 15:48:33 +05:45
Peter Rowlands (변기호) 7a92ff5e5d experiments: commit with --no-verify in executors (#5078)
* git: add no_verify flag for commit()

* experiments: commit with --no-verify in executors

* git: wrap hook exceptions with SCMError

* update hook file perms

* update dulwich to 0.20.15

* update pygit2 commit def

* skip hook tests on windows
2020-12-28 17:35:35 +09:00
Ruslan Kuprieiev 2ce617c642 git: add pygit2 skeleton (#5138)
Implements stuff we need for the tree, but this is more to start filling
up pygit2 with something useful.
2020-12-19 02:44:02 +02:00
João Rafael 84a3a59bc7 Remove version restriction from networkx (#5110) 2020-12-15 19:43:32 +02:00
Peter Rowlands (변기호) 1e52bc94a3 git: add support for working with refspecs and stashes (#4952)
* git: add support for dulwich Stash

* git: start adding support for custom refs

* git: add Stash and helper context managers for preserving
workspace state

* scm: cleanup refs/stash and add push_refspec

* tests: add tests for git refs and stash

* fix push_refspec

* tests: add tests for git push_refspec

* scm: add git fetch-refspecs

* add test for fetch refspecs

* scm: add test for git refs containing

* remove extraneous log msgs

* fix typing issues

* stash: only try to push if workspace is dirty

* update dulwich to 0.20.14

* scm: move dulwich into scm.backend

* move stash internals to backends

* scm: move git code into dvc/scm/git/... module

* git: use ABC for BaseGitBackend

* fix rebase issue

* git: fix refspec tests on windows

- git file:// URLs need posix style paths

* git: move get_refs_containing to backend

* scm: add callback for handling diverged refs during fetch
2020-12-01 16:56:50 +09:00
Saugat Pachhai 6d745eb4f0 s3: use mocked s3 instead of the real one (#4968)
* s3: use mocked s3 instead of the real one

Not removing the fixtures just yet

* Update moto

Tests was failing, because of a change in `responses` package, which
`moto` was mocking. This was fixed on
https://github.com/spulec/moto/pull/3466 which is only available on
`dev122` release.

* Use mocked s3 in checkout tests
2020-11-26 11:30:43 +05:45
Saugat Pachhai 7586c32d95 Remove libs used in pre-commit to avoid duplication (#4966) 2020-11-25 18:30:28 +05:45
Saugat Pachhai d138bdd72a Add mypy config, fix type errors in a few remaining places (#4951)
* Add mypy config, fix type errors in a few random places

* Add typing_extensions

* Use setup.cfg for mypy's config
2020-11-24 14:19:55 +05:45
Saugat Pachhai cb3e52829b interpolation: support [] indices syntax (#4918)
Via pyparsing. The error message is not that great, as it's greedy.
But, it does point out where the mistakes are, correctly most of the
time. :)

Also raises appropriate error messages for ParseError
2020-11-20 14:31:16 +05:45
Ruslan Kuprieiev b340254556 deps: freeze dataclasses version for rich
https://github.com/iterative/dvc/runs/1417985315?check_suite_focus=true
2020-11-18 17:02:22 +02:00
Tomás Farías SantanaandRuslan Kuprieiev ca2e6a004c Support for WebHDFS (#4747)
* Add support for WebHDFS

* Style and linting fixes

* Implement abstract methods and disable no-member pylint errors

* Add init unit test

* Yield client and fix redefinition of hdfs

* Get dynamic hdfs Docker port

* Add support for open in WebHDFSTree

* Remove hdfscli_config fallbacks in WebHDFSTree

* Use port 50070 for webhdfs

* WebHDFSTree supports http URL and proper walking

* Fix failing unit test after refactoring

* Add overwrite argument to upload

* Add remaining func tests for webhdfs

* Remove print statements

* Allow dynamic attributes in pylint for hdfs library

* Comment error on instantiating HDFS client from config

* Remove depth argument from call to walk

* Update dvc/tree/webhdfs.py

* Remove unnecessary deque

* Check exception messages hdfs client failure from config

* Return client directly

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Split long line

* Add size to HashInfo

* Implement move via hdfs.rename

* Add missing line

* Make parent directories before moving file in WebHDFS

* Add total to progress bar

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-11-17 20:02:02 +02:00
Saugat Pachhai 72eb4a5737 Add test server and tests for webdav (#4827)
* Add test server and tests for webdav

* Fix webdav unittests
2020-11-12 13:52:46 +05:45
Sami Jawhar 72ec2a8541 Simpler param updates with python-benedict (#4780) 2020-11-12 16:42:28 +09:00
Ruslan Kuprieiev de01b11483 setup.py: limit responses version for moto (#4879)
```
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\moto\core\models.py:17: in <module>
    import responses
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\responses.py:29: in <module>
    from urllib3.connection import HTTPHeaderDict
E   ImportError: cannot import name 'HTTPHeaderDict'
```
2020-11-11 17:30:27 +02:00