* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* refactor stage into dvcfile
* Split create_stage out of dvcfile to Stage::create() and overwrite
* Remove logging from Repo::run
* tests: stop using --remove-outs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tests: importing from bare repo
* external_repo: don't try too hard to set upstream repo especially if url is not a dvc repo.
* Update tests/func/test_import.py
* Add a comment regarding bare-repo clone
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
Some things are fixed along the way:
- no cache shared between incompatible funcs
- some incosistent exceptions are now consistent
- import updates for git files now works both for dvc and git repos
- "auto-generated-upstream already exists" fixed
- dvc.api.open()/read() works with git repos now
Fixes#3124, #2976. Closes#3172.
* imp: dedup and parallelize external fetches
This prevents same fetches for external artifacts, groups them by source
and fetches in parallel.
* imp: test that fetch works on non workspace deps
* get/import/url: handle existing output directory
Currently if you run
```
dvc get https://github.com/iterative/example-get-started model.pkl --out .
```
it will nuke your cwd. To fix that we need to properly handle existing
directories, same as usual unix cli utilities do that.
* travis: specify python version when installing from chocolatey
Fixes#2609
* ui/perf: do not clone same repo multiple times
Closes#2490
External repos are still created in /tmp. They might be reused until
main() end. If no perfect match is found then a repo with the same url
is copied (not cloned again) and adjusted to fit a request: rev and
cache_dir.
NOTE: test_download_error_pulling_imported_stage() was modified to not
rely on clone error.
* test: fix import update test
* test: fix external_repo test under Windows
* dvc: fix non-master default branch for external repos
* dvc: remove unused .unset_dir() method
* wip - pull is working
* use fetch instead of reproduce
* test pull and fetch
* style changes
* fetch: catch config error on fetch instead of get_remote
* output: remove unused guard clause
* fetch: define a more specific error instead of ConfigError
* fetch: display number of errors correctly
* exceptions: remove "Failed" prefix to download/upload
* fetch: catch other exceptions that might halt the download
* fetch: log catched exceptions
* fetch: reformat exception
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
* fix formatting
* test: clean up test dirs
Closes#1770. Partially does #1888.
* dvc: close anything opened
* test: finalize git repo explicitly to remove test dir later
This was previously an issue on Windows.
Peter Rowlands (변기호)
Paweł Redzyński
Ned Letcher
Casper da Costa-Luis
Ruslan Kuprieiev
Saugat PachhaiandRuslan Kuprieiev
nik123
Mr. Outis
Alexander Schepanovski
Sujith H
Aman Sharma
Chris Kerr
Wojciech Baranowski
Dani Hodovic
Ivan Shcheklein