* Introduce ConfigError and AuthError in dvc.objects.fs and raise it
instead of dvc.config.ConfigError.
* Provide hints and links by catching those errors in dvc.cli.main,
i.e decouple UI messages from filesystems for AuthError and
ConfigError.
* Copy flatten/unflatten utils to dvc.objects.fs.utils.
This commit moves `HashedStreamReader`/`get_hash_file`/`file_md5`
into `dvc.objects.hash`.
The `get_hash_file` has been renamed to `hash_file`.
This also copies `relpath` and `is_exec` to the `dvc.fs.utils`.
Current naming for `DvcFileSystem` and `RepoFileSystem` is confusing, because
dvc is not only about data and it is much more natural to expect something
named `dvcfs` to include complete view for your whole repo (including git
files), which is what `repofs` is doing right now.
`DvcFileSystem` can provide a view not only for the repo data, but also for
external data (e.g. local external outputs). So it makes more sense to call
it a `DataFileSystem`, since it really represents the data management side
of dvc.
This makes dvcfs/repofs much easier to work with and allows it to support
isabs/abspath/etc correctly, as previously any full path (with root_marker "")
would be impossible to distinguish from a relative one.
This also detaches external workspaces from the main repo workspace in
DvcFileSystem, which was necessary to be able to distinguish dvcfs abspaths
from local external output paths. Note that currently we still have limited
support for those (e.g. get/api supports them, but import doesn't) and this
change keeps it that way.
Currently we use both absolute and relative (depend on cwd) paths, which is
terrible for an api, as their interpretation depends on external factors.
This PR makes dvcfs accept only paths relative to the repo root for regular
outputs and absolute paths for external outputs.
Pre-requisite for making repofs use same relative paths.
Introducing simple repo tree that combines all workspaces into one structure,
and allows us to detach data representation from pipelines.
Pre-requisite for dvc-data extraction and using relpaths in dvcfs/repofs.
metadata was introduced before our migration to fsspec and no longer needed
now. This is a pre-requisite to using data index in dvcfs/repofs.
Note that a bunch of awkward flags like isdvc/isout/etc are still kept, and
will be removed in followups.
Initial simple separation to get the process of extracting dvc-objects
and dvc-data packages out of dvc.
dvc-objects is for:
* defining base Object (aka HashFile) and ObjectDB
* providing tools to work with those objects (check/transfer/etc)
* etc
dvc-data is for:
* defining higher level objects (e.g. Tree)
* bulding objects from the workspace (aka staging)
* projecting objects back into the workspace (aka checkout)
* comparing Tree objects (aka diff)
* defining misc object databases (e.g LocalObjectDB)
* higher-level transfer (e.g. with Tree indexes)
* etc
For example, the objects we use in run-cache are based on dvc-objects
and not on dvc-data, as they have absolutely nothing to do with data
management.
There are still some mixups in the code that will be addressed in the
followups.
* test: fix test failure in case of different default branch
When the git config has set global `init.defaultBranch` different than master,
a few test cases were failing.
Currently, the test case is assuming the default branch to be set as
master.
This commit will introduce **initial_branch** option to git init call
which will ensure *master* branch is used as default for out test cases.
Set default git branch to main for test CI
Since main is going to be new default branch, it will let the test case
handle the scenario where default branch is set other than master.
Introduce isolation for git configuration
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.
This PR also removes some redundant/broken remote tests.
Intermediate PR in removing path_info from our code base.
It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.
For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).
Fixes#6774
* objects.status: use object IDs instead of naive objects
* objects.transfer: use object IDs instead of naive objects
* objects: add ReferenceHashFile
* odb: add ReferenceObjectDB
* repo: collect used object IDs instead of naive objs
* support basic/naive fs serialization
* objects.stage: stage file objects in addition to trees
* objects: remove save(), update transfer()
* output/dep: update for osave removal
* path_info: support both url and path attributes
* status/gc: use object IDs instead of naive objs
* diff: update staging usage
* commit: update staging usage
* tests: clean staging when needed
* tests: update for save/transfer/stage changes
* dvcfs: don't allow config serialization (repofs should be used instead)
* utils: return mtime as str, size as int
* tests: update unit tests for save/stage/transfer changes
* objects: use Repo.open when needed in ReferenceHashFile
* fix ls test staging
* objects.stage: don't skip hash check when loading from state
- individual hashes will still be loaded from state if they are still
valid
* ReferenceHashFile: serialize repo fs paths relative to repo root
* preserve state when staging
* objects.tree: skip hash computation in digest() when we have already
loaded hash from state
* always check file existence in referencehashfile
* update repofs test since stage no longer allows nonexistent files
* odb: only protect on check() if hash was verified
* staging: use unique memfs namespace for staging per dest ODB
* tests: clean full staging namespace
* refodb: preserve repofs cache dir
* repodependency: stage imports to local ODB and strict check circular
imports
* fetch: fetch staging last
* staging: make dest odb mandatory and preserve state
* refodb: cache nonlocal fs's
* fix rebase errors
* repofs: support full instantiation from fs config
* repofs: preserve subrepo factory for erepos
* tests: clarify repofs dirty file behavior
Ruslan Kuprieiev
David de la Iglesia Castro
Saugat Pachhai (सौगात)
Daniele Trifirò
Jon Burdo
Amrit GhimireandSaugat Pachhai
Batuhan Taskaya
Nikita Kodenko
Peter Rowlands (변기호)