* support subrepos
This adds support for get/list/api/import/ls on a subrepo.
Also fixes#3180, by adding a granular url support for `dvc.api.get_url`
And, of course it fixes#3369.
* Add tests
* Fix tests, started to get lock_error
* Fix tests for windows
* try to fix mac builds
* Fix comments, readd removed comments
* Fix test
* address review suggestions:
* use `spy` instead of `wraps`
* remove dvc fixture in a test
* set `cache_types` on Git repo
* tests: test importing into subdirectory
* import: fix bug when output subdir doesnt already exist
* use urlparse to determine if output path is local, not os.path.exists
* account for windows drive letters
* import: error out if output subdirectory does not exist
* dvc: get rid of is_walking_tree
No longer needed. Some applications of `isinstance()` will be refactored
in the future, as trees will just provide the needed functionality.
* dvc: move GitTree to dvc/tree/
* dvc: rename *RemoteTree to *Tree
This PR is just a showcase to see how we like it. I've converted a few
tests to it, but there will be much more to come.
```
assert (tmp_dir / "file") == "contents"
assert (tmp_dir / "dir").read_text() == {"subdir": {"file": "hello"}}
```
This approach is nicer because it allows us to use a declaratory
asserts instead of relying on fs files. The problem becomes much more
apparent once you get into remote workspaces, where we `trees_equal`
doesn't work.
* remote: move upload()/download() into tree
* remote: move path_info/path_cls into tree
* remote.azure: finish moving methods into tree
* remote.gs: finish moving methods into tree
* remote.hdfs: finish moving methods to tree
* remote.http: finish moving methods into tree
* remote.oss: finish moving methods into tree
* remote.s3: finish moving methods into tree
* remote.ssh: finish moving methods into tree
* remote.gdrive: finish moving methods into tree
* tests: update remote unit tests
* tests: update func tests
* remote: use walk_files() for all remotes
* list_cache_paths() now uses tree.walk_files() for all remotes except
local/ssh
* fix DS warnings
* bugfixes
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* refactor stage into dvcfile
* Split create_stage out of dvcfile to Stage::create() and overwrite
* Remove logging from Repo::run
* tests: stop using --remove-outs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tests: importing from bare repo
* external_repo: don't try too hard to set upstream repo especially if url is not a dvc repo.
* Update tests/func/test_import.py
* Add a comment regarding bare-repo clone
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
Some things are fixed along the way:
- no cache shared between incompatible funcs
- some incosistent exceptions are now consistent
- import updates for git files now works both for dvc and git repos
- "auto-generated-upstream already exists" fixed
- dvc.api.open()/read() works with git repos now
Fixes#3124, #2976. Closes#3172.
Ioana Grigoropol
GaoandSaugat Pachhai
Kevin Lloyd Bernal
Peter Rowlands (변기호)
Paweł Redzyński
Ned Letcher
Casper da Costa-Luis
Ruslan Kuprieiev
nik123
Mr. Outis
Alexander Schepanovski
Sujith H
Aman Sharma
Chris Kerr
Wojciech Baranowski