Currently, there is no way to find run-cache in HTTP remotes.
In the future, we could use directory-listings from supported
webservers to find files.
Temporarily, we are restricting the run-cache uploads. This is a bit
strict, we could have only restricted the downloads, but it might be
more confusing to the users, so I went to categorically restrict
stage cache uploads and downloads for http filesystems.
We were passing paths with '\\' in the filename instead of posix paths to
remote filesystems which was causing failures to upload.
Although in some filesystems, this was failing silently, so upload would seem
to have passed, but the download later would fail.
`fetch_all_exp()` in `clone()` was called with `url="origin"`, which
resulted in the operation being performed with the remote URL defined
in the cloned repo's config, which did not include any credentials
initially provided to clone.
Fixes#7670
Use relpath to repo.root_dir as keys in deps and outs.
Add `use_cache` and `is_data_source` for outs.
This allows to differentiate git-tracked dependencies and
intermediate outputs.
Closes#7575Closes#7790
This was working for LocalFileSystem as they work for any directories/path.
It's hard to test this these days, as we don't have remote tests inside
dvc, so it was not caught.
Potentially fixes#7718.
The tests should be in dvc-objects repo npw, and will be migrated soon.
The nanotime and diskcache are specified in dvc-objects, so we don't need
those here.
This moves get_fs_config/get_fs_cloud and ODBManager to DVC
as they require DVC related utilities, and we don't need them
strictly in dvc-objects/dvc-data.
This also adds support to ReferenceHashFile to load from
any filesystem, using it's module name and filename.
Now we can get rid of DVCFileSystem dependency in
ReferenceHashFile.
This is a temporary solution, we really need a registry.
(I am not quite sure if this works in pyinstaller mode,
but I think it should, it's how fsspec does it if I understand
correctly.)
transfer raises internal TransferError, merge raises internal MergeError
and local ODB raises ODBError on unprotect.
transfer's error is reraised to FileTransferError back in DataCloud.
This commit:
* defines a `Ignore` protocol in `dvc.objects`, that expects a `find` method.
This will be used in dvc-data and dvc-objects instead of depending on dvc.ignore.
dvcignore is expected to respect this protocol, which it does.
* renames `dvcignore` kwarg to `ignore` in `dvc-data` and `dvc-objects`.
* teaches dvc.cli that `IgnoreInCollectedError` is a known exception
and is logged correctly and adds a test for the behaviour.
I am not happy that we need to hardcode `.dvcignore` in dvc-data
but I don't see any easy way, since we always try to avoid
staging if the tree contains .dvcignore file, even if .dvcignore
is not passed.
Saugat Pachhai (सौगात)
Ruslan Kuprieiev
Daniele Trifirò
dberenbaum
Peter Rowlands
Yury
dependabot[bot]
Paweł Redzyński
huang06
David de la Iglesia Castro
Alexander Schepanovski