* Outputs as target supporting for `dvc status`
fix#4191
1. Add a related test which would fail on current version.
* Pass the tests
1. add deps to the tests.
2. make change to pass the tests.
* Update tests/func/test_status.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Solve some change request.
Co-authored-by: Saugat Pachhai <[email protected]>
* executor: abstract tmpdir population using tree upload
* experiments: unmark flaky test
* fix wrong tree download
* LocalTree: use os.replace instead of os.rename in _upload
* Also fixes plots not getting dumped on flow style
* Make serialization utils aware of trees
* Directly load yamls/tomls using utils, rather than using parser
* use yaml 1.2 everywhere
This commit:
* gets rid of pyyaml
* uses ruamel everywhere
* tries to unite toml and yaml
However, this will require a few more changes that will come soonish.
Eg: Current `parse_yaml` and `parse_toml` behavior could be error-prone,
so, it's better to introduce `tree` inside these serialization methods
(which I will try to tackle next).
Another is, yaml load can be slow for json. We could speed it up just by
using `json.load(s)` here (also in upcoming patch).
* Fix tests
* Imports are no longer moved to top.
* `seed-isort-config` is no longer needed.
* Has a `black`-compatible profile/config.
* Also `isorts` the imports anywhere in the code
(not just at the top).
Also bumps `restyled-isort` to newest version
Currently we kinda assume that whatever is returned by `get_file_hash`
is of type self.PARAM_CHECKSUM, which is not actually true. E.g. for
http it might return `etag` or `md5`, but we don't distinguish between
those and call both `etag`. This is becoming more relevant for dir
hashes that are computed a few different ways (e.g. in-memory md5 or
upload to remote and get etag for the dir file).
Prerequisite for #4144 and #3069
Fix#4333, as `is_ignored_file` and `is_ignored_dir` and `__call__`
share the same call `DvcIgnorePatterns.matches`, we only need to focus
on the out of repo situation.
1. Add tests for the `is_ignored_file` and `is_ignored_dir` for the out
of repo condition.
2. Now `is_ignored_file` and `is_ignored_dir` returns false for those
files outside the repo.
* Remove dvcignore reset in `repo._reset`
Repo `add`,`checkout`,etc will not change `.dvcignore`, and can remove dvcignore reset after these operations.
* Reset dvcignore manually after a subrepo created.
1. subrepo created during session open time.
* tests: add test to ensure dir hashes are not pushed for incomplete
directories
* push: do not push .dir file for directory with missing files
* update test
* Error message when no match found.
1. remove error message when no match found.
* Test update.
1.Test update cover return 0 and 1
* Remove unused arguments
* Error message show in dvc check-ignore
* Allow ignoring subrepos in dvcignore
I implemented this inside dvcignore, and it
should not require "internal" APIs hack as we discussed before.
Right now, `ignore_subrepos` will only work for the paths of subrepo,
not anything inside it. And the API user is more or less only
RepoTree, so should be safe (we switch DvcTree and repo.tree based
on path prefixes, so this is only required during `walk`, so as it
shows the repo that were previously dvcignored and so that we could
switch the trees later during walk.
So, if dir is `dir -> repo`
So, `tree.walk("dir")` will return `[]`,
whereas `tree.walk("dir", ignore_subrepos=False)` will return `["repo"]`
But, successive walk will just return `("repo", [], [])`, and could
just be ignored as a side-effect.
* Fix tests
* Two new mode `-all` and `--stdin` for check-ignore
Fix#4321
Introduce new arguments --all for dvc check-ignore
Add tests for dvc check-ignore --all
* `--stdin` and targets only one
1. `--stdin` and targets can and only can have one
2. more tests
* New tests for stdin mode and solve a problem
1. Add a new tests with mock stdin for interactive mode
2. Solve the problem of `assert "" in caplog.text` in tests always `True`.
* Remove path_input interface
1. remove path_input and make _ask to a public method instead
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* fix styling
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tests: test importing into subdirectory
* import: fix bug when output subdir doesnt already exist
* use urlparse to determine if output path is local, not os.path.exists
* account for windows drive letters
* import: error out if output subdirectory does not exist
Tests had checks for python version number, expecting it to be only
single digits, which was failing for me on 3.6.10 and might fail
in the future with Python 3.10.
I adjusted it to expect minor and patch version to be of multiple digits.
* Update some tests first
fix#3736
* first edition
* solve failure in Windows
* For Windows ci
* Some help ducuments issue.
* Update dvc/ignore.py
abspath
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Refactor with OutOfWorkSpaceError
* Solve a bug
* Update dvc/command/check_ignore.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/check_ignore.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/check_ignore.py
* Revert "Refactor with OutOfWorkSpaceError"
This reverts commit 27eec49ed67346c89157edea998b6b5fa81d88f5.
* Two change request
1. Argument `targets`'s description.
2. Error handling of `_get_normalize_path`
* Update dvc/main.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* `check_ignore` now only accept one path a time
1. Add a new test for the out side repo cases
2. check ignore now only check one file not file lists
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
* tree,remote: add support for webdav
Webdav support is based on https://pypi.org/project/webdavclient3/ and
supports basic download/upload operation, directory creation as well as
existence, file hash and isdir query. Copy, move and remove are also
implemented, though probably not used yet.
WebdavURLInfo is taken from https://github.com/shizacat/dvc/tree/remote-webdav
Fixes iterative#1153
* tree,remote: add further webdavclient3 options
Webdav token auth, certificate and key path and connection timeout are
configurable. Webdav username might be specified or extracted from URL.
Refs iterative#1153
* tree,remote: validate webdav client configuration
Refs iterative#1153
* tree,remote: WebDAV is written with capitalized 'DAV'...
Refs iterative#1153
* tree,remote: terminate WebDAV makedirs at self.path_info.path
This enables the WebDAV api location (e.g. '/public.php/webdav') to be
part of the remote 'url' configuration instead of beeing specified
separately via the 'root' option. The 'root' option may then be used to
specify real directories at the WebDAV storage, although using it to
set the api location is still possible.
Refs iterative#1153
* tree,remote: use >=3.14.5 for WebDAV dependency webdavclient3
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458490609
Refs iterative#1153
* tree,remote: get rid of WebDAV 'root' option and add connection check
The WebDAV 'root' option was rather confusing and should be handled by
the initial 'path_info' from the config 'url' option.
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458754500
While stripping the path/root from the hostname the port got lost, which
is fixed now by simply using the URLInfo 'replace' method as suggested.
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458842853
The WebDAV client connection is tested by probing the existence of the
root (self.path_info.path).
Refs iterative#1153
* tree,remote: implement walk_files for WebDAV
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458751391
Refs: iterative#1153
* tree,remote: let WebDAV client list query file info in walk_files
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458751391
Refs iterative#1153
* tree,remote: add some unit tests for WebDAVTree
Refs iterative#1153
* tree,remote: use ConfigError for WebDAVTree and move Error to webdav.py
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460591937
Refs iterative#1153
* tree,remote: remove/change some (unnecessary) comments
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460592181
* tree,remote: uploading to WebDAV only create directories if not exist
* tree,remote: add BaseTree parameter use_dvcignore to WebDAVTree exists
Refs iterative/#1153
* tree,remote: remove WebDAVTree copy method as proposed
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460892725
Refs iterative#1153
* tree,remote: add progress bar to WebDAV _download and _upload method
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460593130
Refs iterative#1153
Co-authored-by: Christoph Berganski <[email protected]>
* experiments: add initial local tmpdir executor
* fix detached head pull
* experiments: include unchanged (unrepro'd) stages in experiment hash
* experiments: repro single experiment using tmpdir executor by default
* stage experiments as stash commits before running
* include repro args/kwargs in stashed experiments
* support running arbitrary experiment commits (including stash commits)
* experiments: use ProcessPoolExecutor
* experiments: add `dvc exp` alias for `dvc experiments`
* experiments: add `dvc repro --experiment --queue`
* `--queue` can be used to stage an experiment for future execution
* experiments: show queued (unexecuted) experiments in `dvc exp show`
* revert ProcessPoolExecutor change
* experiments: add `dvc repro --run-all [--jobs]`
* `--run-all` can be used to run all queued experiments in parallel
* cleanup output
* experiments: use ProcessPoolExecutor
* fix returning unpicklable objects error
* update tests
* fix windows cleanup issue
* on windows tempdir cannot be removed if we are chdir'd into that
directory
* Version: Redesign output format #3499
Redesigned the output format for the command: dvc version
* version: reformat output format #3499
Done some suggested fixes in code for the dvc command output format issue.
* version: redesign output format #3499
* version: redesign output format for dvc command #3499
* version: redesign output format #3499
* version: redesign output format #3499
* Redesign output format for the dvc version command #3499
* Redesign output format for the dvc version command #3499
* Update dvc/command/version.py
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
karajan1001andSaugat Pachhai
Peter Rowlands (변기호)
Ruslan Kuprieiev
Paweł Redzyński
Puneetha Pai
nik123
AnatolyandRuslan Kuprieiev
Christoph BerganskiandChristoph Berganski
dtrifiroandDaniele Trifirò