If we are asked to open a potentially unitialized repo, we need to make sure
it is not a bare git repo, so that we don't accidentally start putting dvc
files inside git control dir.
Discovered while working on #8789
Stepping stone to simplifying `dvc fetch/pull` by using index.
Fetch handles regular imports through index already, but not repo
imports because their processing is much more involved (e.g. chained
imports) in the current arch.
With `FileStorage` support introduced into `DataIndex` and `datafs`
supporting imports overall, `dvcfs` can now handle repo imports (even
chained ones). This will soon allow us to handle repo imports the same
way we handle regular ones, improve performance and get rid of a lot of
messy code (e.g. DependencyRepo).
Related https://github.com/iterative/scmrepo/issues/207
Related https://github.com/iterative/dvc-data/pull/315
Related https://github.com/iterative/studio/issues/5261
We use all of these for optimization and should not depend on a potentially slow or broken
filesystem on which repository resides (could be HDD or CIFS/NFS/overlayfs/etc).
This PR means a few things:
* all odd filesystems that dvc repo might be on will work quickly out of the box
* when we start caching cloned repositories for `dvc get/import`, we will be
able to cache indexes for them as well (note that bare clones don't have a
worktree so there is no place for .dvc/tmp with the old approach).
* `index.dir` and `state.dir` are now obsolete.
* `dvc init` will cleanup site_cache_dir.
Related #5562
Current code doesn't work for prefixes longer than one, which makes index cache
effectively not work and it should've used has_node from the very start anyway.
Currently we use `open`, which is much slower. For example,
```
$ dvc get dvc-bench data/small/dataset
```
goes down from 110sec to 50sec.
Discovered as a part of https://github.com/iterative/dvc-bench/issues/419
Setting default values is very untypical for all other applications and the
info config has is not enough to make a decision on the default cache dir,
especially when dealing with gitfs.
Fixes#8705
This allows us to reuse cloud versioning index that we've built in other
operations like `data status`.
Note, this only works with `dvc config feature.data_index_cache true`
right now.
Also note that index is not populated for regular remotes, it will come
in a separate PR as there are some http-related issues to figure out.
Related #8962
Extracting from `index.data` as a "root" data index that contains indexes
for different revisions/workspaces.
This is a setup for putting cloud versioning remote indexes and also replacing
old state.
We used to have it named `cache` before, but these days it is clear that
you can have three odbs: just odb (e.g. in-memory), cache (typically local)
or remote (typically cloud). Fixing naming here before proceeding with
introducing `out.odb` and `out.cache`.
In preparation for migration to https://github.com/iterative/dvc-data/pull/304
`data_keys` are similar to `outs`, but they provide more granularity, so we
can use them in `build_data_index` to rebuild index views with filtered
outputs. E.g. when running `dvc diff dir/file`, workspace index should only
have `dir/file` in it.
This also could be used for standalone (non-output) data in the future.
Pre-requisite for #8930
Otherwise when using brancher we end up in a strange situation where even
though we are collecting targets from `gitfs`, we are still using `localfs`
as `self.fs`, which make operations like `self.fs.abspath` in
`collect_granular` return invalid paths.
index: add rev property
This is very useful to identify indexes that are built from git revisions
and so are describing an immutable state and can be used for caching.
We were already using this for `__repr__`.
This is a generally useful function that builds a data index from repo index.
This is currently only used in cloud versioning, but it is going to be used in
`status`, `diff` and other data-centric operations
Cuts down `dvc ls . data/mnist/dataset --recursive` for me from ~25 to ~12
seconds.
`relparts` is an expensive operation as it calls `relpath` that calls `abspath`,
so we want to avoid calling it as much as possible.
Cuts down `dvc ls . data/mnist/dataset --recursive` for me from ~37 to ~25
seconds.
Note that this also changes behaviour for ls with a broken symlink, as we
were not able to see it before because of a failing `info()` call, but now
we can follow a normal UNIX ls behaviour more closely.
stage_infos might contain the same stage multiple times, but with different
filter_info, so we shouldn't convert it to dict, as that makes us lose all
but one filter_info's.
Ruslan Kuprieiev
