* support subrepos
This adds support for get/list/api/import/ls on a subrepo.
Also fixes#3180, by adding a granular url support for `dvc.api.get_url`
And, of course it fixes#3369.
* Add tests
* Fix tests, started to get lock_error
* Fix tests for windows
* try to fix mac builds
* Fix comments, readd removed comments
* Fix test
* address review suggestions:
* use `spy` instead of `wraps`
* remove dvc fixture in a test
* set `cache_types` on Git repo
This PR is just a showcase to see how we like it. I've converted a few
tests to it, but there will be much more to come.
```
assert (tmp_dir / "file") == "contents"
assert (tmp_dir / "dir").read_text() == {"subdir": {"file": "hello"}}
```
This approach is nicer because it allows us to use a declaratory
asserts instead of relying on fs files. The problem becomes much more
apparent once you get into remote workspaces, where we `trees_equal`
doesn't work.
The relative path for external git directory was computed relative to
the root of the repo itself, which created a relative path which is
current working directory. So, all of the contents of the erepo to download
was dumped in the pwd rather than inside of it's own directory.
Fix#4033
* erepo: load external repos directly from git tree
* erepo: add fetch_external
* remote/RepoTree: add callback for keeping track of download counts when
saving tree to cache
* repo: use erepo.fetch_external() in repo.fetch()
* erepo: return cache save_infos in fetch_external
* DependencyRepo: use erepo.fetch_external and cache.checkout instead of
pull_to
* erepo: remove pull_to
* DvcTree: support granularity for isdir()
* DvcTree: fix indentation bug
* RepoTree: add copytree() method
* LocalRemote: use working tree for local cache walk_files
* repo: state should be noop for GitTree based erepos
* erepo: add get_external()
* test_get: remove unneeded caplog check
* erepo: add get_rev()
* tests: fix erepo test bug
* ls: use RepoTree
* remote: support trees in _collect_dir
* remote: support trees in get_dir_checksum
* RepoTree: fix dir_cache relpaths in windows
* remote: fix duplicated fetch bug
* utils: don't skip md5 dos2unix check for git blobs
* otherwise md5 checksums will not match when comparing git version to
cached filesystem version on windows
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
* made '_ignored' check if the given file is already ignored by the root .gitignore. Added corresponding functional test
* code formatting
* Changed method input, adjusted tests, formatted code
* typo
* tests: use git logger fix in conftest
See https://github.com/iterative/dvc/issues/3167
* scm: git: remove redundant comments
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Some things are fixed along the way:
- no cache shared between incompatible funcs
- some incosistent exceptions are now consistent
- import updates for git files now works both for dvc and git repos
- "auto-generated-upstream already exists" fixed
- dvc.api.open()/read() works with git repos now
Fixes#3124, #2976. Closes#3172.
Things changed:
- both no output and no git file result into PathMissingError now
- fail on missing cache or cache download error loudly
- removed sqlite bypass, since we use nolock now anyway
Refactoring:
- streamlined logic and made it similar to Repo.open() one
- moved exclusive exception classes to dvc.repo.get
Saugat Pachhai
Ruslan Kuprieiev
Peter Rowlands (변기호)
Paweł Redzyński
Casper da Costa-Luis
Ruslan Kuprieiev
Mr. Outis
Alexander Schepanovski
Aljoscha SteffensandRuslan Kuprieiev
Jorge Orpinel
Fábio Santos
Aman Sharma
Dani Hodovic