SCM is used for both reading and writing operations, but in the former case -
as filesystem and in the latter - as actual scm/git. So we should distinguish
those and use fs directly, which follows the overall philosophy of being able
to use Repo on any fs.
Keeping `rev=` option for `Repo` because it is a very convenient way to get
repo instance from local path but for particular branch/tag/etc.
Pre-requisite for git/dvc/repofs refactorings.
Intermediate PR in removing path_info from our code base.
It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.
For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).
Fixes#6774
* objects.status: use object IDs instead of naive objects
* objects.transfer: use object IDs instead of naive objects
* objects: add ReferenceHashFile
* odb: add ReferenceObjectDB
* repo: collect used object IDs instead of naive objs
* support basic/naive fs serialization
* objects.stage: stage file objects in addition to trees
* objects: remove save(), update transfer()
* output/dep: update for osave removal
* path_info: support both url and path attributes
* status/gc: use object IDs instead of naive objs
* diff: update staging usage
* commit: update staging usage
* tests: clean staging when needed
* tests: update for save/transfer/stage changes
* dvcfs: don't allow config serialization (repofs should be used instead)
* utils: return mtime as str, size as int
* tests: update unit tests for save/stage/transfer changes
* objects: use Repo.open when needed in ReferenceHashFile
* fix ls test staging
* objects.stage: don't skip hash check when loading from state
- individual hashes will still be loaded from state if they are still
valid
* ReferenceHashFile: serialize repo fs paths relative to repo root
* preserve state when staging
* objects.tree: skip hash computation in digest() when we have already
loaded hash from state
* always check file existence in referencehashfile
* update repofs test since stage no longer allows nonexistent files
* odb: only protect on check() if hash was verified
* staging: use unique memfs namespace for staging per dest ODB
* tests: clean full staging namespace
* refodb: preserve repofs cache dir
* repodependency: stage imports to local ODB and strict check circular
imports
* fetch: fetch staging last
* staging: make dest odb mandatory and preserve state
* refodb: cache nonlocal fs's
* fix rebase errors
* repofs: support full instantiation from fs config
* repofs: preserve subrepo factory for erepos
* tests: clarify repofs dirty file behavior
* objects: add ExternalRepoFile object type
* RepoDependency: implement get_obj (use ExternalRepoFile)
* drop distinction between used external and used objects
* update tests
* remove unneeded fetch from ExternalRepoFile
* wrap erepo errors as an object error
* update cloud/remote functions for new used_objs behavior
* fix update() rev/locked behavior
* update import tests
* remove ExternalRepoFile
* move get_obj and get_used_objs implementations back into repo dependency
* return objects plus associated ODB (remote) in get_used_objs
* odb: add tmp_dir field and support creating Remote from an odb
* return dict mapping odb -> objects in get_used_objs
* use dummy git odb for git imports
* fetch: support pulling from specific odb
* update cloud funcs for get_used_objs usage
* update tests for used_objs behavior
* use fetch in dep.download
* catch circular imports from local fs repos
* add tests for chained and circular imports
* move tmp_dir into odb config
* update return docstring for repo.used_objs
* objects: make HashInfo & HashFile hashable
* stage/output: collect objects (HashFiles) instead of NamedCache
* remote/data_cloud: use objs instead of NamedCache
* repo: use objects instead of NamedCache
* rerun black after rebase
* remove NamedCache usage from run-cache
* drop NamedCache
* update HashInfo.__hash__
* update unit tests
* revert remote_name change
* start cleaning up test failures
* fix tree obj expansion
* update missing cache logging
* fix external dict handling
* update naive odb.gc
* update state save behavior for pull
* update test_data_cloud
* update test_remote
* fix gc with multiple repos
* objects.tree: support filtering w/preserved hash_info
* fix stage cache restore
* drop fs.path_info from HashFile.__hash__
* objects: add optional name attribute
* revert missing cache logging name/suffix changes
* revert name changes in remote tests
* review fixes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update dvc/remote/base.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* state: only load when needed
* state: don't use separate load/dump
* checkout: group link removal logic together
* state: use diskcache.Cache
* staging: use pbar and save to state
* state: don't pass the whole repo to __init__
* staging: explicitly pass state object
* fs: get rid of odb property
* state: don't close the connection to cache
* repo: close: close state
We try to optimize `tree.exists` calls and probably few others
in that they either look directly into the workspace or,
to the cache without running graph checks. It does not seem
to be possible just to run graph checks on `find_outs_by_path`
due to those optimizations.
So, that's why, the `collect` also does a graph check for this
reason.
Fixes#5027Fixes#4010
* state: add simple reentrance
* state: pass Repo instead of LocalCache
* state: use granularly
Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
* addressing: allow stage name addressing without colon
* repo: reorganize imports
* refactor/reorganize collect_granular
* add tests for stage addressing
* fix ds and cc issues
* fix unpack error in test
* tests: stage string representations
* exception: add error message
Also remove `cause` when throwing NoOutputOrStage, as
CheckoutErrorSuggestGit will start chaining all the messages
in cause of the exception.
* rename NoOutputOrStage NoOutputOrStageError
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* repo: separate dir cache and file cache in memory
- `used_cache()`/`get_used_cache()` in repo/stage/output now return
tuples of (dir_cache, file_cache) instead of one flat/merged cache
* update tests for new get_used_cache behavior
* remote: if .dir checksum exists on remote, assume contents also exists
- affects all commands which use `cache_exists()` (remote status)
* push: only upload .dir file after full file contents has been uploaded
* gc: always remove .dir checksums first
* functional tests for push/gc
* repo: support nesting dir caches in NamedCache
* remote: NamedCache updates
* Fix tests
* Fix deepsource warnings
* tests: use pytest mocker fixture
* Update dvc/cache.py
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
* dvc: add Repo._skip_graph_checks flag
Will skip modified graph checks on add, run and import. The aim is to
avoid collecting stages and building graph on these commands.
The skip_checks flag is aimed to be private, but let people use dvc on
big repos without monkey patching or writing yaml files themselves.
* dvc: explain skip_graphs_checks flag in a comment
* test: that skip_graph_checks works
* repo: improve hack description and test
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev
Batuhan Taskaya
Saugat Pachhai
Peter Rowlands (변기호)
Batuhan Taskaya
Alexander Schepanovski
Casper da Costa-Luis
Paweł Redzyński
Mr. Outis