index: data: support imports

Stepping stone to simplifying `dvc fetch/pull` by using index.

Fetch handles regular imports through index already, but not repo
imports because their processing is much more involved (e.g. chained
imports) in the current arch.

With `FileStorage` support introduced into `DataIndex` and `datafs`
supporting imports overall, `dvcfs` can now handle repo imports (even
chained ones). This will soon allow us to handle repo imports the same
way we handle regular ones, improve performance and get rid of a lot of
messy code (e.g. DependencyRepo).

Related https://github.com/iterative/scmrepo/issues/207
Related https://github.com/iterative/dvc-data/pull/315
Related https://github.com/iterative/studio/issues/5261
This commit is contained in:
Ruslan Kuprieiev
2023-03-15 20:32:13 +02:00
parent e6419b939d
commit 53f9953cd1
2 changed files with 8 additions and 1 deletions
+7
View File
@@ -36,11 +36,18 @@ class RepoDependency(Dependency):
}
def __init__(self, def_repo: Dict[str, str], stage: "Stage", *args, **kwargs):
from dvc.fs import DVCFileSystem
self.def_repo = def_repo
self._objs: Dict[str, "HashFile"] = {}
self._meta: Dict[str, "Meta"] = {}
super().__init__(stage, *args, **kwargs)
self.fs = DVCFileSystem(
self.def_repo[self.PARAM_URL],
rev=self.def_repo.get(self.PARAM_REV_LOCK),
)
def _parse_path(self, fs, fs_path): # noqa: ARG002
return None
+1 -1
View File
@@ -161,7 +161,7 @@ def _load_storage_from_out(storage_map, key, out):
except NoRemoteError:
pass
if out.stage.is_import and not out.stage.is_repo_import:
if out.stage.is_import:
dep = out.stage.deps[0]
storage_map.add_data(FileStorage(key, dep.fs, dep.fs_path))