Makes dvc get/import/etc use dvcfs, which already supports cloud versioning,
circular imports and other stuff.
Also makes dvc import behave more like dvc import-url, so that we can use
the same existing logic for fetching those using index instead of objects.
Fixes#8789
Related iterative/studio#4782
Currently if dvc_dir doesn't exists (e.g. we are creating an uninitialized
repo), config will try looking for `.dvc` dir in the current directory,
which might be inside of another dvc repository resulting in mixed configs.
Use explicit `from_cwd` method to make `Config` try to find repo configs
if available to avoid the usage mixup.
* erepo: get rid of NoRemoteInExternalRepoError
This exception hasn't been making any sense at all since we've introduced
get and import, as they can be chained in repos.
* erepo: get rid of NoOutputInExternalRepoError
This is dead code these days.
* erepo: get rid of FileMissingError and PathMissingError
* api: preserve legacy exc wrappers
If we are asked to open a potentially unitialized repo, we need to make sure
it is not a bare git repo, so that we don't accidentally start putting dvc
files inside git control dir.
Discovered while working on #8789
Stepping stone to simplifying `dvc fetch/pull` by using index.
Fetch handles regular imports through index already, but not repo
imports because their processing is much more involved (e.g. chained
imports) in the current arch.
With `FileStorage` support introduced into `DataIndex` and `datafs`
supporting imports overall, `dvcfs` can now handle repo imports (even
chained ones). This will soon allow us to handle repo imports the same
way we handle regular ones, improve performance and get rid of a lot of
messy code (e.g. DependencyRepo).
Related https://github.com/iterative/scmrepo/issues/207
Related https://github.com/iterative/dvc-data/pull/315
Related https://github.com/iterative/studio/issues/5261
We use all of these for optimization and should not depend on a potentially slow or broken
filesystem on which repository resides (could be HDD or CIFS/NFS/overlayfs/etc).
This PR means a few things:
* all odd filesystems that dvc repo might be on will work quickly out of the box
* when we start caching cloned repositories for `dvc get/import`, we will be
able to cache indexes for them as well (note that bare clones don't have a
worktree so there is no place for .dvc/tmp with the old approach).
* `index.dir` and `state.dir` are now obsolete.
* `dvc init` will cleanup site_cache_dir.
Related #5562
Current code doesn't work for prefixes longer than one, which makes index cache
effectively not work and it should've used has_node from the very start anyway.
Currently we use `open`, which is much slower. For example,
```
$ dvc get dvc-bench data/small/dataset
```
goes down from 110sec to 50sec.
Discovered as a part of https://github.com/iterative/dvc-bench/issues/419
Setting default values is very untypical for all other applications and the
info config has is not enough to make a decision on the default cache dir,
especially when dealing with gitfs.
Fixes#8705
This allows us to reuse cloud versioning index that we've built in other
operations like `data status`.
Note, this only works with `dvc config feature.data_index_cache true`
right now.
Also note that index is not populated for regular remotes, it will come
in a separate PR as there are some http-related issues to figure out.
Related #8962
Extracting from `index.data` as a "root" data index that contains indexes
for different revisions/workspaces.
This is a setup for putting cloud versioning remote indexes and also replacing
old state.
We used to have it named `cache` before, but these days it is clear that
you can have three odbs: just odb (e.g. in-memory), cache (typically local)
or remote (typically cloud). Fixing naming here before proceeding with
introducing `out.odb` and `out.cache`.
In preparation for migration to https://github.com/iterative/dvc-data/pull/304
`data_keys` are similar to `outs`, but they provide more granularity, so we
can use them in `build_data_index` to rebuild index views with filtered
outputs. E.g. when running `dvc diff dir/file`, workspace index should only
have `dir/file` in it.
This also could be used for standalone (non-output) data in the future.
Pre-requisite for #8930
Otherwise when using brancher we end up in a strange situation where even
though we are collecting targets from `gitfs`, we are still using `localfs`
as `self.fs`, which make operations like `self.fs.abspath` in
`collect_granular` return invalid paths.
index: add rev property
This is very useful to identify indexes that are built from git revisions
and so are describing an immutable state and can be used for caching.
We were already using this for `__repr__`.
This is a generally useful function that builds a data index from repo index.
This is currently only used in cloud versioning, but it is going to be used in
`status`, `diff` and other data-centric operations
Cuts down `dvc ls . data/mnist/dataset --recursive` for me from ~25 to ~12
seconds.
`relparts` is an expensive operation as it calls `relpath` that calls `abspath`,
so we want to avoid calling it as much as possible.
Cuts down `dvc ls . data/mnist/dataset --recursive` for me from ~37 to ~25
seconds.
Note that this also changes behaviour for ls with a broken symlink, as we
were not able to see it before because of a failing `info()` call, but now
we can follow a normal UNIX ls behaviour more closely.
stage_infos might contain the same stage multiple times, but with different
filter_info, so we shouldn't convert it to dict, as that makes us lose all
but one filter_info's.
Ruslan Kuprieiev
