Current naming for `DvcFileSystem` and `RepoFileSystem` is confusing, because
dvc is not only about data and it is much more natural to expect something
named `dvcfs` to include complete view for your whole repo (including git
files), which is what `repofs` is doing right now.
`DvcFileSystem` can provide a view not only for the repo data, but also for
external data (e.g. local external outputs). So it makes more sense to call
it a `DataFileSystem`, since it really represents the data management side
of dvc.
Our gitfs paths are currently tied to the local path in which the cloned
repository resides in, which results in a lot of hacky assumptions throughout
the code.
This PR detaches gitfs from local paths completely, making dvc work with a
proper fsspec gitfs implementation.
Part of #7543
This makes dvcfs/repofs much easier to work with and allows it to support
isabs/abspath/etc correctly, as previously any full path (with root_marker "")
would be impossible to distinguish from a relative one.
This also detaches external workspaces from the main repo workspace in
DvcFileSystem, which was necessary to be able to distinguish dvcfs abspaths
from local external output paths. Note that currently we still have limited
support for those (e.g. get/api supports them, but import doesn't) and this
change keeps it that way.
Currently we use both absolute and relative (depend on cwd) paths, which is
terrible for an api, as their interpretation depends on external factors.
This PR makes dvcfs accept only paths relative to the repo root for regular
outputs and absolute paths for external outputs.
Pre-requisite for making repofs use same relative paths.
Introducing simple repo tree that combines all workspaces into one structure,
and allows us to detach data representation from pipelines.
Pre-requisite for dvc-data extraction and using relpaths in dvcfs/repofs.
Using outs directly breaks isolation and makes it really hard to understand
what's going on.
Redundant abspaths will go away in followups when moving to relpaths.
metadata was introduced before our migration to fsspec and no longer needed
now. This is a pre-requisite to using data index in dvcfs/repofs.
Note that a bunch of awkward flags like isdvc/isout/etc are still kept, and
will be removed in followups.
With our staging process automatically capturing file metadata, we no longer
need to set isexec ourselves.
In followup PRs we will also use meta during checkout, so that we won't need
to use Output.set_exec by hand.
mypy:
```
dvc/utils/table.py: note: In member "add_column" of class "Table":
dvc/utils/table.py:24:5: error: Argument 7 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['top'], Literal['middle'], Literal['bottom']]" [override]
def add_column( # pylint: disable=arguments-differ
^
dvc/utils/table.py:24:5: note: This violates the Liskov substitution principle
dvc/utils/table.py:24:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
dvc/utils/table.py:24:5: error: Argument 8 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['fold'], Literal['crop'], Literal['ellipsis'], Literal['ignore']]"
[override]
def add_column( # pylint: disable=arguments-differ
^
dvc/utils/table.py:24:5: error: Argument 12 of "add_column" is incompatible
with supertype "Table"; supertype defines the argument type as "Optional[int]"
[override]
def add_column( # pylint: disable=arguments-differ
^
Found 3 errors in 1 file (checked 129 source files)
dvc/ui/table.py:87: note: In module imported here,
dvc/compare.py:25: note: ... from here:
dvc/utils/table.py: note: In member "add_column" of class "Table":
dvc/utils/table.py:24:5: error: Argument 7 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['top'], Literal['middle'], Literal['bottom']]" [override]
def add_column( # pylint: disable=arguments-differ
^
dvc/utils/table.py:24:5: note: This violates the Liskov substitution principle
dvc/utils/table.py:24:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
dvc/utils/table.py:24:5: error: Argument 8 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['fold'], Literal['crop'], Literal['ellipsis'], Literal['ignore']]"
[override]
def add_column( # pylint: disable=arguments-differ
^
dvc/utils/table.py:24:5: error: Argument 12 of "add_column" is incompatible
with supertype "Table"; supertype defines the argument type as "Optional[int]"
[override]
def add_column( # pylint: disable=arguments-differ
^
Found 3 errors in 1 file (checked 128 source files)
```
pylint:
```
************* Module dvc.utils.table
dvc/utils/table.py:24:4: W0237: Parameter 'vertical' has been renamed to 'overflow' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'overflow' has been renamed to 'width' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'width' has been renamed to 'min_width' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'min_width' has been renamed to 'max_width' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'max_width' has been renamed to 'ratio' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'ratio' has been renamed to 'no_wrap' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'no_wrap' has been renamed to 'collapse' in overridden 'Table.add_column' method (arguments-renamed)
```
Initial simple separation to get the process of extracting dvc-objects
and dvc-data packages out of dvc.
dvc-objects is for:
* defining base Object (aka HashFile) and ObjectDB
* providing tools to work with those objects (check/transfer/etc)
* etc
dvc-data is for:
* defining higher level objects (e.g. Tree)
* bulding objects from the workspace (aka staging)
* projecting objects back into the workspace (aka checkout)
* comparing Tree objects (aka diff)
* defining misc object databases (e.g LocalObjectDB)
* higher-level transfer (e.g. with Tree indexes)
* etc
For example, the objects we use in run-cache are based on dvc-objects
and not on dvc-data, as they have absolutely nothing to do with data
management.
There are still some mixups in the code that will be addressed in the
followups.
SCM is used for both reading and writing operations, but in the former case -
as filesystem and in the latter - as actual scm/git. So we should distinguish
those and use fs directly, which follows the overall philosophy of being able
to use Repo on any fs.
Keeping `rev=` option for `Repo` because it is a very convenient way to get
repo instance from local path but for particular branch/tag/etc.
Pre-requisite for git/dvc/repofs refactorings.
These options are part of the default config for each particular remote.
Plugins will define their default config on theirown, but for now we keep
those in one config schema.
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.
This PR also removes some redundant/broken remote tests.
This is a very early version of test utils for dvc plugins. This module
consists of existing tests with minimal changes and is going to evolve
a lot as our plugins are getting filled in and pytest-test-utils matures.
All three of those use more or less the same logic for testing different
link types and maybe then using them. So now instead of having 3
implementations, we'll have just one.
Filesystem `transfer` now will not try to catch particular errno's, but
instead will just ignore any OSError and move down to the next link type
to try out, and if the last one fails - it will raise the last exception,
so if it is a common problem - we'll be able to see it.
This patch also removes `odb.confirmed_cache_type`, as it was caching
link type that is not really a cache-level thing, but rather a checkout thing,
that should only be cached for particular objects we are linking, as multiple
objects could be linked to different filesystems from the same odb (e.g.
external local output on hdd and local data on your ssd).
Next step here will be to introduce metadata diff, so that we can get rid of
temporary Link wrapper and introduce Transfer and corresponding transfer
config, that will know what links to use and could have some other things
like callbacks, so we don't continue passing dozens of kwargs along the stack.
Continuing from #6963Fixes#6980
Currently whether or not link confirmation will run depends on
particular filesystem and existing workspace link type, which introduces
inconsistency in behaviour, where we sometimes catch unsupported links
before touching user data and sometimes we don't.
Depending on a filesyste (e.g. on NTFS), `_remove` might reset
read-only permissions in order to delete a hardlink to protected object,
which will also reset it for the object itself, making it unprotected,
so we need to protect it back.
Currently it depends on the whole `info()` output, which has too much
information that doesn't really matter. To check if file contents didn't
change mtime, size, inode combo is enough.
If supported, reflink, will provide the fastest transfer possible.
That being said, right now we use `move` in most of the places,
which still takes presidence.
Pre-requisite for #6387
* dvc: drop 3.6 support
Considering that 3.6 has ~2 months until EOL, and our deps are
dropping support for it already (e.g. pygit2), we are droppipping
it as well. This will reduce the load on our test CI as well, since
we've started supporting 3.10 recently.
* setup.cfg: adjust requirements
* daemon: remove 3.6 logic
* tests: utils: remove pre 3.7 logic
Ruslan Kuprieiev