This avoids recollecting the cache files over and over again.
For example, `dvc fetch` noop in `dvc-bench` went from ~30sec down to ~10sec
for me.
Next step is to also cache remote index, so we don't have to waste those
~10sec checking files in the remote.
Related https://github.com/iterative/dvc-data/pull/376
This ensures we don't assume that everything needs to have hash_info to be
checked out (e.g. we can totally checkout directly from FileStorage).
Related https://github.com/iterative/dvc/pull/9424
Makes dvc get/import/etc use dvcfs, which already supports cloud versioning,
circular imports and other stuff.
Also makes dvc import behave more like dvc import-url, so that we can use
the same existing logic for fetching those using index instead of objects.
Fixes#8789
Related iterative/studio#4782
Currently if dvc_dir doesn't exists (e.g. we are creating an uninitialized
repo), config will try looking for `.dvc` dir in the current directory,
which might be inside of another dvc repository resulting in mixed configs.
Use explicit `from_cwd` method to make `Config` try to find repo configs
if available to avoid the usage mixup.
* erepo: get rid of NoRemoteInExternalRepoError
This exception hasn't been making any sense at all since we've introduced
get and import, as they can be chained in repos.
* erepo: get rid of NoOutputInExternalRepoError
This is dead code these days.
* erepo: get rid of FileMissingError and PathMissingError
* api: preserve legacy exc wrappers
If we are asked to open a potentially unitialized repo, we need to make sure
it is not a bare git repo, so that we don't accidentally start putting dvc
files inside git control dir.
Discovered while working on #8789
Stepping stone to simplifying `dvc fetch/pull` by using index.
Fetch handles regular imports through index already, but not repo
imports because their processing is much more involved (e.g. chained
imports) in the current arch.
With `FileStorage` support introduced into `DataIndex` and `datafs`
supporting imports overall, `dvcfs` can now handle repo imports (even
chained ones). This will soon allow us to handle repo imports the same
way we handle regular ones, improve performance and get rid of a lot of
messy code (e.g. DependencyRepo).
Related https://github.com/iterative/scmrepo/issues/207
Related https://github.com/iterative/dvc-data/pull/315
Related https://github.com/iterative/studio/issues/5261
We use all of these for optimization and should not depend on a potentially slow or broken
filesystem on which repository resides (could be HDD or CIFS/NFS/overlayfs/etc).
This PR means a few things:
* all odd filesystems that dvc repo might be on will work quickly out of the box
* when we start caching cloned repositories for `dvc get/import`, we will be
able to cache indexes for them as well (note that bare clones don't have a
worktree so there is no place for .dvc/tmp with the old approach).
* `index.dir` and `state.dir` are now obsolete.
* `dvc init` will cleanup site_cache_dir.
Related #5562
Current code doesn't work for prefixes longer than one, which makes index cache
effectively not work and it should've used has_node from the very start anyway.
Currently we use `open`, which is much slower. For example,
```
$ dvc get dvc-bench data/small/dataset
```
goes down from 110sec to 50sec.
Discovered as a part of https://github.com/iterative/dvc-bench/issues/419
Setting default values is very untypical for all other applications and the
info config has is not enough to make a decision on the default cache dir,
especially when dealing with gitfs.
Fixes#8705
This allows us to reuse cloud versioning index that we've built in other
operations like `data status`.
Note, this only works with `dvc config feature.data_index_cache true`
right now.
Also note that index is not populated for regular remotes, it will come
in a separate PR as there are some http-related issues to figure out.
Related #8962
Extracting from `index.data` as a "root" data index that contains indexes
for different revisions/workspaces.
This is a setup for putting cloud versioning remote indexes and also replacing
old state.
We used to have it named `cache` before, but these days it is clear that
you can have three odbs: just odb (e.g. in-memory), cache (typically local)
or remote (typically cloud). Fixing naming here before proceeding with
introducing `out.odb` and `out.cache`.
In preparation for migration to https://github.com/iterative/dvc-data/pull/304
`data_keys` are similar to `outs`, but they provide more granularity, so we
can use them in `build_data_index` to rebuild index views with filtered
outputs. E.g. when running `dvc diff dir/file`, workspace index should only
have `dir/file` in it.
This also could be used for standalone (non-output) data in the future.
Pre-requisite for #8930
Ruslan Kuprieiev