We used to have it named `cache` before, but these days it is clear that
you can have three odbs: just odb (e.g. in-memory), cache (typically local)
or remote (typically cloud). Fixing naming here before proceeding with
introducing `out.odb` and `out.cache`.
In preparation for migration to https://github.com/iterative/dvc-data/pull/304
* cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc.
* remove pylint direcitves
* Apply suggestions from code review
* Update pyproject.toml
This adds support for custom metadata in outputs under `meta` key.
Also adds `--meta` flag to `add`/`import`/`import-url` that supports
`key=value` values.
This moves get_fs_config/get_fs_cloud and ODBManager to DVC
as they require DVC related utilities, and we don't need them
strictly in dvc-objects/dvc-data.
This also adds support to ReferenceHashFile to load from
any filesystem, using it's module name and filename.
Now we can get rid of DVCFileSystem dependency in
ReferenceHashFile.
This is a temporary solution, we really need a registry.
(I am not quite sure if this works in pyinstaller mode,
but I think it should, it's how fsspec does it if I understand
correctly.)
This commit moves `HashedStreamReader`/`get_hash_file`/`file_md5`
into `dvc.objects.hash`.
The `get_hash_file` has been renamed to `hash_file`.
This also copies `relpath` and `is_exec` to the `dvc.fs.utils`.
It is not completely fsspec compliant, but it's upto the level
required by our FSSpecWrapper.
Next thing to tackle would be to merging `fs.base.FileSystem`
and `fs.fsspec_wrapper.FSSpecWrapper` into one filesystem
that is lazily-loaded and is itself fsspec-compliant.
By splitting dvc.cli into a module, we'll be able to share cli-related
utilities, split parser-related stuffs into smaller parts, improve
reusability and improve API for command extensions in the future.
Initial simple separation to get the process of extracting dvc-objects
and dvc-data packages out of dvc.
dvc-objects is for:
* defining base Object (aka HashFile) and ObjectDB
* providing tools to work with those objects (check/transfer/etc)
* etc
dvc-data is for:
* defining higher level objects (e.g. Tree)
* bulding objects from the workspace (aka staging)
* projecting objects back into the workspace (aka checkout)
* comparing Tree objects (aka diff)
* defining misc object databases (e.g LocalObjectDB)
* higher-level transfer (e.g. with Tree indexes)
* etc
For example, the objects we use in run-cache are based on dvc-objects
and not on dvc-data, as they have absolutely nothing to do with data
management.
There are still some mixups in the code that will be addressed in the
followups.
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.
This PR also removes some redundant/broken remote tests.
This is a very early version of test utils for dvc plugins. This module
consists of existing tests with minimal changes and is going to evolve
a lot as our plugins are getting filled in and pytest-test-utils matures.
skshetry
dberenbaum
Ruslan Kuprieiev
Oded Messer
Karan Jakhar
David de la Iglesia Castro
Saugat Pachhai (सौगात)
Ronan Lamy
Daniele Trifirò
Karthikeyan Singaravelan
Samuel Knight
Zhenyu Gao