* tree: validate resolved remote:// sections
* dvc: use checksum_jobs instead of hash_jobs
This is a regression introduced during one of the refactorings.
* Print dvc version info for debugging
fix#4095
1. seperate info from dvc version.
2. print dvc version info when some unexpected error occured.
* Psutil NoneType error
* Update tests/func/test_version.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Move `package` outside dvc/utils/pkg.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
The API of `Dvcfile.dump` has been changed to
`dump(stage, update_pipeline=True, update_lock=True)`,
which will make it to always dump by default to both files,
and provide flags to suppress one or the other.
Fixes#4235
* allow for a config option for http method
* add tests
* fix tests with method
* Update dvc/tree/http.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
We used to use that logic in DvcTree back before we had proper trees.
Now we have a RepoTree, that combines git/workspace tree with the
DvcTree and dirty state clearly belongs to the git/workspace and not
the dvc itself, as dvc state only depends on dvc-files and lock-files.
Prerequisite for #4523
* diff: support missing cache status for workspace diff
* diff: add --hide-missing option to hide cache missing status
* update command tests
* update diff func tests
* only show missing cache status when diffing local workspace
* suppress all output if no changes and --hide-missing
* use hide_missing instead of show_missing
* use is_dvc
* dvc: override replace in HTTPURLInfo to include extra_parts
The objective is to passtrough any parameter configured in http remotes.
fixes#4508
* add unit test for #4508
* Use HTTPURLInfo directly instead of decorator
* support subrepos
This adds support for get/list/api/import/ls on a subrepo.
Also fixes#3180, by adding a granular url support for `dvc.api.get_url`
And, of course it fixes#3369.
* Add tests
* Fix tests, started to get lock_error
* Fix tests for windows
* try to fix mac builds
* Fix comments, readd removed comments
* Fix test
* address review suggestions:
* use `spy` instead of `wraps`
* remove dvc fixture in a test
* set `cache_types` on Git repo
By itself `self.info` is quite confusing, as it is not clear what it is
about. Using `hash_info` makes much more sense and is required to
support alternative hash types.
Related to #4144, #3069, #1676
* experiments: cleanup show UI
* show git tag/branch names instead of SHAs by default
* make timestamp formatting more like the viewer
* update tests
* place timestamps in their own column
* update command test
* exp: remove unused regex
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: use cache state when saving downloaded cache
Related to #4485
* tree: dont compute hash from scratch for output dirs
Fixes#4485
* tree: fix get_hash
* plots: add -e/--experiment option for plots diff
* experiments: add method to get baseline for a given experiment
* plots: default to diff against baseline instead of workspace/head for experiments
* use exact match for experiments RE since name-rev may return relative
commits
* diff: Add sort logic to sort diff ouput irrespective of output format(default/json/md)
* diff: test stub using actual value, not pass by reference for (dict)
* use yaml 1.2 everywhere
This commit:
* gets rid of pyyaml
* uses ruamel everywhere
* tries to unite toml and yaml
However, this will require a few more changes that will come soonish.
Eg: Current `parse_yaml` and `parse_toml` behavior could be error-prone,
so, it's better to introduce `tree` inside these serialization methods
(which I will try to tackle next).
Another is, yaml load can be slow for json. We could speed it up just by
using `json.load(s)` here (also in upcoming patch).
* Fix tests
* Imports are no longer moved to top.
* `seed-isort-config` is no longer needed.
* Has a `black`-compatible profile/config.
* Also `isorts` the imports anywhere in the code
(not just at the top).
Also bumps `restyled-isort` to newest version
Currently we kinda assume that whatever is returned by `get_file_hash`
is of type self.PARAM_CHECKSUM, which is not actually true. E.g. for
http it might return `etag` or `md5`, but we don't distinguish between
those and call both `etag`. This is becoming more relevant for dir
hashes that are computed a few different ways (e.g. in-memory md5 or
upload to remote and get etag for the dir file).
Prerequisite for #4144 and #3069
* Update some tests first
fix#3736
* first edition
* solve failure in Windows
* For Windows ci
* Some help ducuments issue.
* Update dvc/ignore.py
abspath
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Refactor with OutOfWorkSpaceError
* Solve a bug
* Update dvc/command/check_ignore.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/check_ignore.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/check_ignore.py
* Revert "Refactor with OutOfWorkSpaceError"
This reverts commit 27eec49ed67346c89157edea998b6b5fa81d88f5.
* Two change request
1. Argument `targets`'s description.
2. Error handling of `_get_normalize_path`
* Update dvc/main.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* `check_ignore` now only accept one path a time
1. Add a new test for the out side repo cases
2. check ignore now only check one file not file lists
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
* tree,remote: add support for webdav
Webdav support is based on https://pypi.org/project/webdavclient3/ and
supports basic download/upload operation, directory creation as well as
existence, file hash and isdir query. Copy, move and remove are also
implemented, though probably not used yet.
WebdavURLInfo is taken from https://github.com/shizacat/dvc/tree/remote-webdav
Fixes iterative#1153
* tree,remote: add further webdavclient3 options
Webdav token auth, certificate and key path and connection timeout are
configurable. Webdav username might be specified or extracted from URL.
Refs iterative#1153
* tree,remote: validate webdav client configuration
Refs iterative#1153
* tree,remote: WebDAV is written with capitalized 'DAV'...
Refs iterative#1153
* tree,remote: terminate WebDAV makedirs at self.path_info.path
This enables the WebDAV api location (e.g. '/public.php/webdav') to be
part of the remote 'url' configuration instead of beeing specified
separately via the 'root' option. The 'root' option may then be used to
specify real directories at the WebDAV storage, although using it to
set the api location is still possible.
Refs iterative#1153
* tree,remote: use >=3.14.5 for WebDAV dependency webdavclient3
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458490609
Refs iterative#1153
* tree,remote: get rid of WebDAV 'root' option and add connection check
The WebDAV 'root' option was rather confusing and should be handled by
the initial 'path_info' from the config 'url' option.
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458754500
While stripping the path/root from the hostname the port got lost, which
is fixed now by simply using the URLInfo 'replace' method as suggested.
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458842853
The WebDAV client connection is tested by probing the existence of the
root (self.path_info.path).
Refs iterative#1153
* tree,remote: implement walk_files for WebDAV
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458751391
Refs: iterative#1153
* tree,remote: let WebDAV client list query file info in walk_files
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458751391
Refs iterative#1153
* tree,remote: add some unit tests for WebDAVTree
Refs iterative#1153
* tree,remote: use ConfigError for WebDAVTree and move Error to webdav.py
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460591937
Refs iterative#1153
* tree,remote: remove/change some (unnecessary) comments
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460592181
* tree,remote: uploading to WebDAV only create directories if not exist
* tree,remote: add BaseTree parameter use_dvcignore to WebDAVTree exists
Refs iterative/#1153
* tree,remote: remove WebDAVTree copy method as proposed
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460892725
Refs iterative#1153
* tree,remote: add progress bar to WebDAV _download and _upload method
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460593130
Refs iterative#1153
Co-authored-by: Christoph Berganski <[email protected]>
* experiments: add initial local tmpdir executor
* fix detached head pull
* experiments: include unchanged (unrepro'd) stages in experiment hash
* experiments: repro single experiment using tmpdir executor by default
* stage experiments as stash commits before running
* include repro args/kwargs in stashed experiments
* support running arbitrary experiment commits (including stash commits)
* experiments: use ProcessPoolExecutor
* experiments: add `dvc exp` alias for `dvc experiments`
* experiments: add `dvc repro --experiment --queue`
* `--queue` can be used to stage an experiment for future execution
* experiments: show queued (unexecuted) experiments in `dvc exp show`
* revert ProcessPoolExecutor change
* experiments: add `dvc repro --run-all [--jobs]`
* `--run-all` can be used to run all queued experiments in parallel
* cleanup output
* experiments: use ProcessPoolExecutor
* fix returning unpicklable objects error
* update tests
* fix windows cleanup issue
* on windows tempdir cannot be removed if we are chdir'd into that
directory
* Update
1. unit test update
2. two might cause bugs
* dvc ignore
* Some refactoring
* clean
* Config tree use
* Solve sub dir repo problem. Add more tests
* typo error
* Change request
Co-authored-by: karajan1001 <[email protected]>
* dvc: get rid of is_walking_tree
No longer needed. Some applications of `isinstance()` will be refactored
in the future, as trees will just provide the needed functionality.
* dvc: move GitTree to dvc/tree/
* dvc: rename *RemoteTree to *Tree
* key api
* Restyled by black
* test
* fixes
* test
* Restyled by black
* Trigger CI
* default val
* remove redundant default
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update tests/unit/remote/test_s3.py
* rename fields
* update test
* key_id->secret_key_id
Co-authored-by: Restyled.io <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* experiments: add initial experiments.show
* only includes metrics + params (no code/data information)
* experiments: basic `dvc experiments show` functionality
* experiments: clone and run repro inside clone workspace
* repro: add -e/--experiment option
* experiments: hash experiments to identify duplicates
* experiments: update show
* experiments: checkout experiment after running repro
* experiments: add `dvc experiments checkout`
* experiments: add simple `dvc experiments diff` command
* just shows combined output from `metrics diff` and `plots diff` for
now
* add very simple tests
* add some useful output messages
* experiments: suppress help messages while feature is in development
* experiment related commands will be accessible but hidden from the
default command help messages
* add simple experiments command tests
* experiments: disable feature by default
* experiments only enabled in test environment (DVC_TEST) or when
core.experiments config option is true
* use fetch on checkout missing revs
* dvcignore: move file/dir/parent checks into DvcIgnoreFilter
* dvc: get rid of CleanTree
CleanTree is a very awkward wrapper that spreads the context across
the codebase and results in unexpected behaviour when using it.
This PR starts moving dvcignore-related logic into the trees themselves
(it makes a lot of sense, kinda like `state`) so they could deal with
it however they like.
There are at least two temporary ugly parts about this PR:
1) dvcignore is used by individual trees and not packed into
tree/base.py yet;
2) `dvcignore_root` argument. This one is caused by the dvcignore trying
to collect everything topdown starting from the certain root dir. What
it should do instead is for a certain path that is being checked look up
the tree through the parents until it finds repo root (.dvc dir) and
then stop. That would handle subrepos as well. At the same time we need
to leverage existing dvcignore trie structure to cache those results.
Working tree is really just a regular local tree and should be
used by outputs when trying to compute a hash for themselves.
We didn't use it previously because local tree was embeded into
the local remote class.
Related to #4050
Ruslan Kuprieiev
karajan1001andRuslan Kuprieiev
Saugat Pachhai
Robert F. DickersonandRuslan Kuprieiev
Victor
Alexander Schepanovski
Peter Rowlands (변기호)
Puneetha Pai
Paweł Redzyński
Christoph BerganskiandChristoph Berganski
dtrifiroandDaniele Trifirò
