* git: add support for dulwich Stash
* git: start adding support for custom refs
* git: add Stash and helper context managers for preserving
workspace state
* scm: cleanup refs/stash and add push_refspec
* tests: add tests for git refs and stash
* fix push_refspec
* tests: add tests for git push_refspec
* scm: add git fetch-refspecs
* add test for fetch refspecs
* scm: add test for git refs containing
* remove extraneous log msgs
* fix typing issues
* stash: only try to push if workspace is dirty
* update dulwich to 0.20.14
* scm: move dulwich into scm.backend
* move stash internals to backends
* scm: move git code into dvc/scm/git/... module
* git: use ABC for BaseGitBackend
* fix rebase issue
* git: fix refspec tests on windows
- git file:// URLs need posix style paths
* git: move get_refs_containing to backend
* scm: add callback for handling diverged refs during fetch
* s3: use mocked s3 instead of the real one
Not removing the fixtures just yet
* Update moto
Tests was failing, because of a change in `responses` package, which
`moto` was mocking. This was fixed on
https://github.com/spulec/moto/pull/3466 which is only available on
`dev122` release.
* Use mocked s3 in checkout tests
Via pyparsing. The error message is not that great, as it's greedy.
But, it does point out where the mistakes are, correctly most of the
time. :)
Also raises appropriate error messages for ParseError
* Add support for WebHDFS
* Style and linting fixes
* Implement abstract methods and disable no-member pylint errors
* Add init unit test
* Yield client and fix redefinition of hdfs
* Get dynamic hdfs Docker port
* Add support for open in WebHDFSTree
* Remove hdfscli_config fallbacks in WebHDFSTree
* Use port 50070 for webhdfs
* WebHDFSTree supports http URL and proper walking
* Fix failing unit test after refactoring
* Add overwrite argument to upload
* Add remaining func tests for webhdfs
* Remove print statements
* Allow dynamic attributes in pylint for hdfs library
* Comment error on instantiating HDFS client from config
* Remove depth argument from call to walk
* Update dvc/tree/webhdfs.py
* Remove unnecessary deque
* Check exception messages hdfs client failure from config
* Return client directly
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Split long line
* Add size to HashInfo
* Implement move via hdfs.rename
* Add missing line
* Make parent directories before moving file in WebHDFS
* Add total to progress bar
Co-authored-by: Ruslan Kuprieiev <[email protected]>
```
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\moto\core\models.py:17: in <module>
import responses
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\responses.py:29: in <module>
from urllib3.connection import HTTPHeaderDict
E ImportError: cannot import name 'HTTPHeaderDict'
```
moto has aws-sam-translator as an indirect dependency that since 1.29.0
started to have a public package
that causes conflicts in flufl.lock when it tries to use atpublic
package.
```
tests/func/test_add.py:21: in <module>
from dvc.main import main
dvc/main.py:7: in <module>
from dvc import analytics
dvc/analytics.py:16: in <module>
from dvc.lock import Lock, LockError
dvc/lock.py:8: in <module>
import flufl.lock
/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/flufl/lock/__init__.py:3: in <module>
from flufl.lock._lockfile import (
/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/flufl/lock/_lockfile.py:42: in <module>
from public import public
E ImportError: cannot import name 'public'
```
Oddly enough, this seems to also been causing pylint issues. public is a
terrible name for a package SMH
In my tests `Git._ignore` is taking around ~65seconds for the whole test
suit(~50K invocations). The reason is that `gitpython` is launching a real
`git check-ignore` process, that takes time and resources.
Switching to `pygit2` or `dulwich` makes it drop to ~5seconds, saving us
almost a minute of the test run time and a lot of resources by not
launching any processes. I've chosen `dulwich` over `pygit2` here
because it is much more pythonic and as long as the performance is
on-par with `pygit2` it is better for us to use `dulwich`.
* Run CI for Python3.9, skips hdfs tests for 3.9
Also, `dvc[hdfs]` install will be a no-op till we have
a pyarrow wheel for 3.9.
Note that this does not add an entry to the classifier
claiming it supports 3.9, that will be added when 3.9
gets released (or, wait for pyarrow to be released).
* Skip HDFS related tests on Python3.9
Also, fix the python version regex for `3.9.0b5` like versions.
* Exclude python3.9 for Windows
libxml does not have wheels for Python39.
We do compile them on Linux and Mac
* Update tests/unit/test_info.py
* Use realpath for gitignore
Windows tests for Python3.8 was failing
intermittently.
Note: Also removed the realpath wrap around self.root_dir as it's already a
realpath when `Git` is instantiated.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Paramiko has recently added support for `Match`
https://github.com/paramiko/paramiko/issues/717
which resulted in some users getting an import error about `invoke`
module, as it is not a default dependecy for paramiko.
@Pseudomanifold reported that installing `invoke` #4589 worked for
him, which means that this newly added paramiko feature works in
his case. That feature seems to not affect configs without `Match`,
so we could give this a try and install it by default.
Our conda package will be updated during the next release.
Fixes#4589Fixes#4616
* use yaml 1.2 everywhere
This commit:
* gets rid of pyyaml
* uses ruamel everywhere
* tries to unite toml and yaml
However, this will require a few more changes that will come soonish.
Eg: Current `parse_yaml` and `parse_toml` behavior could be error-prone,
so, it's better to introduce `tree` inside these serialization methods
(which I will try to tackle next).
Another is, yaml load can be slow for json. We could speed it up just by
using `json.load(s)` here (also in upcoming patch).
* Fix tests
Ruslan Kuprieiev
Peter Rowlands (변기호)
Saugat Pachhai
Asif Saif UddinandRuslan Kuprieiev
João Rafael
Tomás Farías SantanaandRuslan Kuprieiev
Sami Jawhar
Tomás Farías Santana
omar.zhouandzwh
Casper da Costa-Luis
David Roberts
Paweł Redzyński