* Run CI for Python3.9, skips hdfs tests for 3.9
Also, `dvc[hdfs]` install will be a no-op till we have
a pyarrow wheel for 3.9.
Note that this does not add an entry to the classifier
claiming it supports 3.9, that will be added when 3.9
gets released (or, wait for pyarrow to be released).
* Skip HDFS related tests on Python3.9
Also, fix the python version regex for `3.9.0b5` like versions.
* Exclude python3.9 for Windows
libxml does not have wheels for Python39.
We do compile them on Linux and Mac
* Update tests/unit/test_info.py
* Use realpath for gitignore
Windows tests for Python3.8 was failing
intermittently.
Note: Also removed the realpath wrap around self.root_dir as it's already a
realpath when `Git` is instantiated.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Paramiko has recently added support for `Match`
https://github.com/paramiko/paramiko/issues/717
which resulted in some users getting an import error about `invoke`
module, as it is not a default dependecy for paramiko.
@Pseudomanifold reported that installing `invoke` #4589 worked for
him, which means that this newly added paramiko feature works in
his case. That feature seems to not affect configs without `Match`,
so we could give this a try and install it by default.
Our conda package will be updated during the next release.
Fixes#4589Fixes#4616
* use yaml 1.2 everywhere
This commit:
* gets rid of pyyaml
* uses ruamel everywhere
* tries to unite toml and yaml
However, this will require a few more changes that will come soonish.
Eg: Current `parse_yaml` and `parse_toml` behavior could be error-prone,
so, it's better to introduce `tree` inside these serialization methods
(which I will try to tackle next).
Another is, yaml load can be slow for json. We could speed it up just by
using `json.load(s)` here (also in upcoming patch).
* Fix tests
* tree,remote: add support for webdav
Webdav support is based on https://pypi.org/project/webdavclient3/ and
supports basic download/upload operation, directory creation as well as
existence, file hash and isdir query. Copy, move and remove are also
implemented, though probably not used yet.
WebdavURLInfo is taken from https://github.com/shizacat/dvc/tree/remote-webdav
Fixes iterative#1153
* tree,remote: add further webdavclient3 options
Webdav token auth, certificate and key path and connection timeout are
configurable. Webdav username might be specified or extracted from URL.
Refs iterative#1153
* tree,remote: validate webdav client configuration
Refs iterative#1153
* tree,remote: WebDAV is written with capitalized 'DAV'...
Refs iterative#1153
* tree,remote: terminate WebDAV makedirs at self.path_info.path
This enables the WebDAV api location (e.g. '/public.php/webdav') to be
part of the remote 'url' configuration instead of beeing specified
separately via the 'root' option. The 'root' option may then be used to
specify real directories at the WebDAV storage, although using it to
set the api location is still possible.
Refs iterative#1153
* tree,remote: use >=3.14.5 for WebDAV dependency webdavclient3
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458490609
Refs iterative#1153
* tree,remote: get rid of WebDAV 'root' option and add connection check
The WebDAV 'root' option was rather confusing and should be handled by
the initial 'path_info' from the config 'url' option.
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458754500
While stripping the path/root from the hostname the port got lost, which
is fixed now by simply using the URLInfo 'replace' method as suggested.
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458842853
The WebDAV client connection is tested by probing the existence of the
root (self.path_info.path).
Refs iterative#1153
* tree,remote: implement walk_files for WebDAV
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458751391
Refs: iterative#1153
* tree,remote: let WebDAV client list query file info in walk_files
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458751391
Refs iterative#1153
* tree,remote: add some unit tests for WebDAVTree
Refs iterative#1153
* tree,remote: use ConfigError for WebDAVTree and move Error to webdav.py
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460591937
Refs iterative#1153
* tree,remote: remove/change some (unnecessary) comments
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460592181
* tree,remote: uploading to WebDAV only create directories if not exist
* tree,remote: add BaseTree parameter use_dvcignore to WebDAVTree exists
Refs iterative/#1153
* tree,remote: remove WebDAVTree copy method as proposed
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460892725
Refs iterative#1153
* tree,remote: add progress bar to WebDAV _download and _upload method
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460593130
Refs iterative#1153
Co-authored-by: Christoph Berganski <[email protected]>
https://github.com/pytest-dev/pytest/issues/7558
pylint throws error for `pytest.mark.*` functions in pytest6.
```
tests/unit/repo/test_repo.py:15:1: E1102: pytest.mark.parametrize is not callable (not-callable)
tests/unit/repo/test_repo.py:33:1: E1102: pytest.mark.parametrize is not callable (not-callable)
************* Module tests.unit.remote.ssh.test_connection
tests/unit/remote/ssh/test_connection.py:87:1: E1102: pytest.mark.skipif is not callable (not-callable)
tests/unit/remote/ssh/test_connection.py:100:1: E1102: pytest.mark.skipif is not callable (not-callable)
```
https://github.com/iterative/dvc/pull/4257#issuecomment-665474467
```
pylint...................................................................Failed
- hook id: pylint
- exit code: 2
Registered custom plugin. Some checks will be disabled for tests.
************* Module dvc.command.experiments
dvc/command/experiments.py:14:4: E0401: Unable to import 'flatten_json'
(import-error)
dvc/command/experiments.py:39:4: E0401: Unable to import 'flatten_json'
(import-error)
-----------------------------------
Your code has been rated at 9.99/10
Registered custom plugin. Some checks will be disabled for tests.
************* Module dvc.utils.diff
dvc/utils/diff.py:38:8: E0401: Unable to import 'flatten_json'
(import-error)
************* Module dvc.command.metrics
dvc/command/metrics.py:17:4: E0401: Unable to import 'flatten_json'
(import-error)
-----------------------------------
```
* experiments: add initial experiments.show
* only includes metrics + params (no code/data information)
* experiments: basic `dvc experiments show` functionality
* experiments: clone and run repro inside clone workspace
* repro: add -e/--experiment option
* experiments: hash experiments to identify duplicates
* experiments: update show
* experiments: checkout experiment after running repro
* experiments: add `dvc experiments checkout`
* experiments: add simple `dvc experiments diff` command
* just shows combined output from `metrics diff` and `plots diff` for
now
* add very simple tests
* add some useful output messages
* experiments: suppress help messages while feature is in development
* experiment related commands will be accessible but hidden from the
default command help messages
* add simple experiments command tests
* experiments: disable feature by default
* experiments only enabled in test environment (DVC_TEST) or when
core.experiments config option is true
* use fetch on checkout missing revs
flufl.lock released 4.0 in which there is no finalize method anymore, hence
```
dvc/lock.py:143:16: E1101: Instance of 'HardlinkLock' has no 'finalize' member (no-member)
```
* pylint: enable checks for imports
* ci: add github action for pylint
* enable pylint on pre-commit
* Install requirements
* use local pylint installation rather than of pre-commit's
* fix import errors
* more checks
* Silence unused imports
* Make a copy of dict
* remove unnecessary generators/comprehensions
* Use dict literals
* remove unused variables
* misc fixes
ignore .iteritems, is a false +ve
remove function call in default argument
* Flake8 config: complexity, show source on err, show err counts
* flake8: add extensions
* remove flake8-string-format
* Add flake8-debugger support
* Add flake8-string-format extension
* remove flake8-debugger
It's having some problems to install on Windows in Travis
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
* params/metrics: diff: implement tabulated markdown output
Via `--show-md` on `diff`s.
* table: show None on missing val
* tests: use `dedent`, add tests for markdown
* diff: add --show-md
Co-authored-by: Ruslan Kuprieiev <[email protected]>
omar.zhouandzwh
Saugat PachhaiandRuslan Kuprieiev
Casper da Costa-Luis
David Roberts
Paweł Redzyński
Christoph BerganskiandChristoph Berganski
dtrifiroandDaniele Trifirò
Peter Rowlands (변기호)
faucct
Ivan Shcheklein