* gdrive: minor clean using funcy 1.14 features
Employ @retry(filter_errors) and @wrap_prop() to pack things up a bit.
* gdrive: refactor dirs/ids/root_id cache and their uses
- cache root transparently
- make cache dirs and ids symmetric, i.e. both using full paths
- implement .list_cache_paths() instead of .all() like in the rest of
the remotes
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
* erepo: cache all read only external repos by hexsha
So we have 3 things cached now separately:
- clean clones, to not ask for creds repatedly
- cache dirs, also shared between erepos with same origin
- checked out clones if they are read only, addressed by (url, hexsha)
Several additions to `Git` along the way:
- Git.is_sha() static method
- .pull() and .push() work with multiple returned records correctly
- .get_rev() and .resolve_rev() work faster
- .resolve_rev() looks for remote branches
- .has_rev()
Fixes#3280.
* git: improve .resolve_rev()
It follows `git checkout` logic now - if name can be unambiguously
resolved across known remotes then it's done.
- code moving during git pull won't affect any erepos
- source tag move will work as expected
- source cache dir setting will never break havoc
- cache is reused between all clones from same source
Some things are fixed along the way:
- no cache shared between incompatible funcs
- some incosistent exceptions are now consistent
- import updates for git files now works both for dvc and git repos
- "auto-generated-upstream already exists" fixed
- dvc.api.open()/read() works with git repos now
Fixes#3124, #2976. Closes#3172.
- proper doc-string for prepare_summon()
- summon -> summon_dict in summon() to not overwrite a glo bal name.
This was not a bug, but a bad practice, which might have caused some
issues in the future.
- better name for _get_object_spec()
Things changed:
- both no output and no git file result into PathMissingError now
- fail on missing cache or cache download error loudly
- removed sqlite bypass, since we use nolock now anyway
Refactoring:
- streamlined logic and made it similar to Repo.open() one
- moved exclusive exception classes to dvc.repo.get
- `tmp_dir` descends from Path
- convenient template methods to generate files and dirs, add them to
dvc and git and commit to git
- independent of old basic_env structures
- can go in any order
- modular: start with empty `tmp_dir`, initialize git with `scm`, dvc
repo with `dvc` add "repo template" with `repo_template`, ...
With @cached_property last write wins, so all threads will use the same
session. However, several might be created concurrently wasting
resources and even potentially asking something from the user.
This change guarantees only one client instance will be created for a
remote.
This permites URLInfo construction from parts without building/reparsing
string, which makes it around 20% faster. Code is also simpler and a bit
shorter now.
Other improvements:
- added URLInfo.replace(path=...)
- URLInfo._path now stringifies properly
- protected against passing path not starting with /
- do not allow query, params, fragments and passwords in urls (they were
silently ignored previously)
* imp: dedup and parallelize external fetches
This prevents same fetches for external artifacts, groups them by source
and fetches in parallel.
* imp: test that fetch works on non workspace deps
* dvc: fix branch suffixes in status/push/pull
* gc: add --all-commits flag
* gc: fix gc warning
Say that we always leave things used in the working tree
* dvc: dedup used cache
This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts
* dvc: delay checksum names concatenation
* ui/perf: do not clone same repo multiple times
Closes#2490
External repos are still created in /tmp. They might be reused until
main() end. If no perfect match is found then a repo with the same url
is copied (not cloned again) and adjusted to fit a request: rev and
cache_dir.
NOTE: test_download_error_pulling_imported_stage() was modified to not
rely on clone error.
* test: fix import update test
* test: fix external_repo test under Windows
* dvc: fix non-master default branch for external repos
* dvc: remove unused .unset_dir() method
This avoids importing `pkg_resources` on startup, which saves around 200
ms.
NOTE: wheel installer uses its own entry points template, which is the same
fastentrypoints.py script provides, so no gain there.
