Our gitfs paths are currently tied to the local path in which the cloned
repository resides in, which results in a lot of hacky assumptions throughout
the code.
This PR detaches gitfs from local paths completely, making dvc work with a
proper fsspec gitfs implementation.
Part of #7543
This makes dvcfs/repofs much easier to work with and allows it to support
isabs/abspath/etc correctly, as previously any full path (with root_marker "")
would be impossible to distinguish from a relative one.
This also detaches external workspaces from the main repo workspace in
DvcFileSystem, which was necessary to be able to distinguish dvcfs abspaths
from local external output paths. Note that currently we still have limited
support for those (e.g. get/api supports them, but import doesn't) and this
change keeps it that way.
fix: #7448
Current we skip on those experiments which already existed on both sides, but
do not exclude them from the result list. And if we have diverged experiment,
pulling/pushing them will stop the current progress and raise an exception, but
didn't give any information on the condition of other experiments.
1. Change the exp pull and push to match the new API in `scmrepo`.
2. Handle sync status after `exp pull/push` finished.
3. Bump scmrepo to 0.0.19
Co-authored-by:
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.
The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.
This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.
The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.
This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
It is not completely fsspec compliant, but it's upto the level
required by our FSSpecWrapper.
Next thing to tackle would be to merging `fs.base.FileSystem`
and `fs.fsspec_wrapper.FSSpecWrapper` into one filesystem
that is lazily-loaded and is itself fsspec-compliant.
Also adds a kwarg `cancel_on_error` on the constructor to support
cancelling futures on `shutdown`. This is not done by default to
be backward compatible with `concurrent.futures.ThreadPool` executor.
Karthikeyan Singaravelan
Ruslan Kuprieiev
Saugat Pachhai (सौगात)
karajan1001
dberenbaum
Samuel Knight
Paweł Redzyński
David de la Iglesia Castro
Peter Rowlands
dependabot[bot]
Nick SorrosandPaweł Redzyński