Relpath might not exist (e.g. on windows between different drives),
we should just use path as is and not get too fancy with the ui,
unless we want to constantly keep ValueError in mind when using
relpath for ui needs all over the place.
These operations should use index, where we always have current
directory and relative paths, so the ui will change accordingly
anyway.
* output: save: error out on cached external outputs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
With all fetch and checkout related things that we've done in the recent past,
we now have all the mechanisms necessary to provide actual progress (no total,
but still better than spinner) during `dvc data status` operations.
* fs: callbacks: create our own TqdmCallback
We have a near identical implementation in dvc-objects, but this one is
meant to work with our Tqdm progress instance.
dvc-objects one is useful on other levels, but dvc should have itsown
that inherits from dvc-objects' Callback.
* progress: add as_callback
* checkout: use ui.progress
The build process is much more stable these days and there is no reason
to keep building these packages here. Instead every package has itsown
building process (dvc-osxpkg, dvc-exe, dvc-s3-repo) and knows best how
to build and sign correctly.
pyinstaller hooks need to go into a `__pyinstaller` later on.
For some unknown reason the mock in this test leaks into the outside tests.
I wasn't able to determine why in the reasonable amount of time, so instead just
simplifying it to not use mocker at all.
That whole file should really be migrated to dvc-data, but that needs some time.
Had the "pleasure" of finding a mocker.patch leak into other tests and one
of the suspicions is that mixing unittest.mock and mocker might be somehow
related.
In any case, we should use mocker where possible anyway.
This avoids recollecting the cache files over and over again.
For example, `dvc fetch` noop in `dvc-bench` went from ~30sec down to ~10sec
for me.
Next step is to also cache remote index, so we don't have to waste those
~10sec checking files in the remote.
Related https://github.com/iterative/dvc-data/pull/376
This ensures we don't assume that everything needs to have hash_info to be
checked out (e.g. we can totally checkout directly from FileStorage).
Related https://github.com/iterative/dvc/pull/9424
Makes dvc get/import/etc use dvcfs, which already supports cloud versioning,
circular imports and other stuff.
Also makes dvc import behave more like dvc import-url, so that we can use
the same existing logic for fetching those using index instead of objects.
Fixes#8789
Related iterative/studio#4782
Currently if dvc_dir doesn't exists (e.g. we are creating an uninitialized
repo), config will try looking for `.dvc` dir in the current directory,
which might be inside of another dvc repository resulting in mixed configs.
Use explicit `from_cwd` method to make `Config` try to find repo configs
if available to avoid the usage mixup.
* erepo: get rid of NoRemoteInExternalRepoError
This exception hasn't been making any sense at all since we've introduced
get and import, as they can be chained in repos.
* erepo: get rid of NoOutputInExternalRepoError
This is dead code these days.
* erepo: get rid of FileMissingError and PathMissingError
* api: preserve legacy exc wrappers
If we are asked to open a potentially unitialized repo, we need to make sure
it is not a bare git repo, so that we don't accidentally start putting dvc
files inside git control dir.
Discovered while working on #8789
Stepping stone to simplifying `dvc fetch/pull` by using index.
Fetch handles regular imports through index already, but not repo
imports because their processing is much more involved (e.g. chained
imports) in the current arch.
With `FileStorage` support introduced into `DataIndex` and `datafs`
supporting imports overall, `dvcfs` can now handle repo imports (even
chained ones). This will soon allow us to handle repo imports the same
way we handle regular ones, improve performance and get rid of a lot of
messy code (e.g. DependencyRepo).
Related https://github.com/iterative/scmrepo/issues/207
Related https://github.com/iterative/dvc-data/pull/315
Related https://github.com/iterative/studio/issues/5261
We use all of these for optimization and should not depend on a potentially slow or broken
filesystem on which repository resides (could be HDD or CIFS/NFS/overlayfs/etc).
This PR means a few things:
* all odd filesystems that dvc repo might be on will work quickly out of the box
* when we start caching cloned repositories for `dvc get/import`, we will be
able to cache indexes for them as well (note that bare clones don't have a
worktree so there is no place for .dvc/tmp with the old approach).
* `index.dir` and `state.dir` are now obsolete.
* `dvc init` will cleanup site_cache_dir.
Related #5562
Current code doesn't work for prefixes longer than one, which makes index cache
effectively not work and it should've used has_node from the very start anyway.
Currently we use `open`, which is much slower. For example,
```
$ dvc get dvc-bench data/small/dataset
```
goes down from 110sec to 50sec.
Discovered as a part of https://github.com/iterative/dvc-bench/issues/419
Ruslan Kuprieiev
