* git: add support for dulwich Stash
* git: start adding support for custom refs
* git: add Stash and helper context managers for preserving
workspace state
* scm: cleanup refs/stash and add push_refspec
* tests: add tests for git refs and stash
* fix push_refspec
* tests: add tests for git push_refspec
* scm: add git fetch-refspecs
* add test for fetch refspecs
* scm: add test for git refs containing
* remove extraneous log msgs
* fix typing issues
* stash: only try to push if workspace is dirty
* update dulwich to 0.20.14
* scm: move dulwich into scm.backend
* move stash internals to backends
* scm: move git code into dvc/scm/git/... module
* git: use ABC for BaseGitBackend
* fix rebase issue
* git: fix refspec tests on windows
- git file:// URLs need posix style paths
* git: move get_refs_containing to backend
* scm: add callback for handling diverged refs during fetch
* s3: use mocked s3 instead of the real one
Not removing the fixtures just yet
* Update moto
Tests was failing, because of a change in `responses` package, which
`moto` was mocking. This was fixed on
https://github.com/spulec/moto/pull/3466 which is only available on
`dev122` release.
* Use mocked s3 in checkout tests
Via pyparsing. The error message is not that great, as it's greedy.
But, it does point out where the mistakes are, correctly most of the
time. :)
Also raises appropriate error messages for ParseError
* Add support for WebHDFS
* Style and linting fixes
* Implement abstract methods and disable no-member pylint errors
* Add init unit test
* Yield client and fix redefinition of hdfs
* Get dynamic hdfs Docker port
* Add support for open in WebHDFSTree
* Remove hdfscli_config fallbacks in WebHDFSTree
* Use port 50070 for webhdfs
* WebHDFSTree supports http URL and proper walking
* Fix failing unit test after refactoring
* Add overwrite argument to upload
* Add remaining func tests for webhdfs
* Remove print statements
* Allow dynamic attributes in pylint for hdfs library
* Comment error on instantiating HDFS client from config
* Remove depth argument from call to walk
* Update dvc/tree/webhdfs.py
* Remove unnecessary deque
* Check exception messages hdfs client failure from config
* Return client directly
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Split long line
* Add size to HashInfo
* Implement move via hdfs.rename
* Add missing line
* Make parent directories before moving file in WebHDFS
* Add total to progress bar
Co-authored-by: Ruslan Kuprieiev <[email protected]>
```
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\moto\core\models.py:17: in <module>
import responses
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\responses.py:29: in <module>
from urllib3.connection import HTTPHeaderDict
E ImportError: cannot import name 'HTTPHeaderDict'
```
moto has aws-sam-translator as an indirect dependency that since 1.29.0
started to have a public package
that causes conflicts in flufl.lock when it tries to use atpublic
package.
```
tests/func/test_add.py:21: in <module>
from dvc.main import main
dvc/main.py:7: in <module>
from dvc import analytics
dvc/analytics.py:16: in <module>
from dvc.lock import Lock, LockError
dvc/lock.py:8: in <module>
import flufl.lock
/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/flufl/lock/__init__.py:3: in <module>
from flufl.lock._lockfile import (
/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/flufl/lock/_lockfile.py:42: in <module>
from public import public
E ImportError: cannot import name 'public'
```
Oddly enough, this seems to also been causing pylint issues. public is a
terrible name for a package SMH
In my tests `Git._ignore` is taking around ~65seconds for the whole test
suit(~50K invocations). The reason is that `gitpython` is launching a real
`git check-ignore` process, that takes time and resources.
Switching to `pygit2` or `dulwich` makes it drop to ~5seconds, saving us
almost a minute of the test run time and a lot of resources by not
launching any processes. I've chosen `dulwich` over `pygit2` here
because it is much more pythonic and as long as the performance is
on-par with `pygit2` it is better for us to use `dulwich`.
* Run CI for Python3.9, skips hdfs tests for 3.9
Also, `dvc[hdfs]` install will be a no-op till we have
a pyarrow wheel for 3.9.
Note that this does not add an entry to the classifier
claiming it supports 3.9, that will be added when 3.9
gets released (or, wait for pyarrow to be released).
* Skip HDFS related tests on Python3.9
Also, fix the python version regex for `3.9.0b5` like versions.
* Exclude python3.9 for Windows
libxml does not have wheels for Python39.
We do compile them on Linux and Mac
* Update tests/unit/test_info.py
* Use realpath for gitignore
Windows tests for Python3.8 was failing
intermittently.
Note: Also removed the realpath wrap around self.root_dir as it's already a
realpath when `Git` is instantiated.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Paramiko has recently added support for `Match`
https://github.com/paramiko/paramiko/issues/717
which resulted in some users getting an import error about `invoke`
module, as it is not a default dependecy for paramiko.
@Pseudomanifold reported that installing `invoke` #4589 worked for
him, which means that this newly added paramiko feature works in
his case. That feature seems to not affect configs without `Match`,
so we could give this a try and install it by default.
Our conda package will be updated during the next release.
Fixes#4589Fixes#4616
* use yaml 1.2 everywhere
This commit:
* gets rid of pyyaml
* uses ruamel everywhere
* tries to unite toml and yaml
However, this will require a few more changes that will come soonish.
Eg: Current `parse_yaml` and `parse_toml` behavior could be error-prone,
so, it's better to introduce `tree` inside these serialization methods
(which I will try to tackle next).
Another is, yaml load can be slow for json. We could speed it up just by
using `json.load(s)` here (also in upcoming patch).
* Fix tests
* tree,remote: add support for webdav
Webdav support is based on https://pypi.org/project/webdavclient3/ and
supports basic download/upload operation, directory creation as well as
existence, file hash and isdir query. Copy, move and remove are also
implemented, though probably not used yet.
WebdavURLInfo is taken from https://github.com/shizacat/dvc/tree/remote-webdav
Fixes iterative#1153
* tree,remote: add further webdavclient3 options
Webdav token auth, certificate and key path and connection timeout are
configurable. Webdav username might be specified or extracted from URL.
Refs iterative#1153
* tree,remote: validate webdav client configuration
Refs iterative#1153
* tree,remote: WebDAV is written with capitalized 'DAV'...
Refs iterative#1153
* tree,remote: terminate WebDAV makedirs at self.path_info.path
This enables the WebDAV api location (e.g. '/public.php/webdav') to be
part of the remote 'url' configuration instead of beeing specified
separately via the 'root' option. The 'root' option may then be used to
specify real directories at the WebDAV storage, although using it to
set the api location is still possible.
Refs iterative#1153
* tree,remote: use >=3.14.5 for WebDAV dependency webdavclient3
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458490609
Refs iterative#1153
* tree,remote: get rid of WebDAV 'root' option and add connection check
The WebDAV 'root' option was rather confusing and should be handled by
the initial 'path_info' from the config 'url' option.
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458754500
While stripping the path/root from the hostname the port got lost, which
is fixed now by simply using the URLInfo 'replace' method as suggested.
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458842853
The WebDAV client connection is tested by probing the existence of the
root (self.path_info.path).
Refs iterative#1153
* tree,remote: implement walk_files for WebDAV
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458751391
Refs: iterative#1153
* tree,remote: let WebDAV client list query file info in walk_files
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458751391
Refs iterative#1153
* tree,remote: add some unit tests for WebDAVTree
Refs iterative#1153
* tree,remote: use ConfigError for WebDAVTree and move Error to webdav.py
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460591937
Refs iterative#1153
* tree,remote: remove/change some (unnecessary) comments
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460592181
* tree,remote: uploading to WebDAV only create directories if not exist
* tree,remote: add BaseTree parameter use_dvcignore to WebDAVTree exists
Refs iterative/#1153
* tree,remote: remove WebDAVTree copy method as proposed
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460892725
Refs iterative#1153
* tree,remote: add progress bar to WebDAV _download and _upload method
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460593130
Refs iterative#1153
Co-authored-by: Christoph Berganski <[email protected]>
https://github.com/pytest-dev/pytest/issues/7558
pylint throws error for `pytest.mark.*` functions in pytest6.
```
tests/unit/repo/test_repo.py:15:1: E1102: pytest.mark.parametrize is not callable (not-callable)
tests/unit/repo/test_repo.py:33:1: E1102: pytest.mark.parametrize is not callable (not-callable)
************* Module tests.unit.remote.ssh.test_connection
tests/unit/remote/ssh/test_connection.py:87:1: E1102: pytest.mark.skipif is not callable (not-callable)
tests/unit/remote/ssh/test_connection.py:100:1: E1102: pytest.mark.skipif is not callable (not-callable)
```
https://github.com/iterative/dvc/pull/4257#issuecomment-665474467
Ruslan Kuprieiev
Asif Saif UddinandRuslan Kuprieiev
Saugat Pachhai
Peter Rowlands (변기호)
João Rafael
Tomás Farías SantanaandRuslan Kuprieiev
Sami Jawhar
Tomás Farías Santana
omar.zhouandzwh
Casper da Costa-Luis
David Roberts
Paweł Redzyński
Christoph BerganskiandChristoph Berganski