By splitting dvc.cli into a module, we'll be able to share cli-related
utilities, split parser-related stuffs into smaller parts, improve
reusability and improve API for command extensions in the future.
Initial simple separation to get the process of extracting dvc-objects
and dvc-data packages out of dvc.
dvc-objects is for:
* defining base Object (aka HashFile) and ObjectDB
* providing tools to work with those objects (check/transfer/etc)
* etc
dvc-data is for:
* defining higher level objects (e.g. Tree)
* bulding objects from the workspace (aka staging)
* projecting objects back into the workspace (aka checkout)
* comparing Tree objects (aka diff)
* defining misc object databases (e.g LocalObjectDB)
* higher-level transfer (e.g. with Tree indexes)
* etc
For example, the objects we use in run-cache are based on dvc-objects
and not on dvc-data, as they have absolutely nothing to do with data
management.
There are still some mixups in the code that will be addressed in the
followups.
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.
This PR also removes some redundant/broken remote tests.
This is a very early version of test utils for dvc plugins. This module
consists of existing tests with minimal changes and is going to evolve
a lot as our plugins are getting filled in and pytest-test-utils matures.
* output: use remote kwarg instead of odb
Fixes#6596
* Update dvc/output.py
Co-authored-by: Dave Berenbaum <[email protected]>
* add test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Dave Berenbaum <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* objects.status: use object IDs instead of naive objects
* objects.transfer: use object IDs instead of naive objects
* objects: add ReferenceHashFile
* odb: add ReferenceObjectDB
* repo: collect used object IDs instead of naive objs
* support basic/naive fs serialization
* objects.stage: stage file objects in addition to trees
* objects: remove save(), update transfer()
* output/dep: update for osave removal
* path_info: support both url and path attributes
* status/gc: use object IDs instead of naive objs
* diff: update staging usage
* commit: update staging usage
* tests: clean staging when needed
* tests: update for save/transfer/stage changes
* dvcfs: don't allow config serialization (repofs should be used instead)
* utils: return mtime as str, size as int
* tests: update unit tests for save/stage/transfer changes
* objects: use Repo.open when needed in ReferenceHashFile
* fix ls test staging
* objects.stage: don't skip hash check when loading from state
- individual hashes will still be loaded from state if they are still
valid
* ReferenceHashFile: serialize repo fs paths relative to repo root
* preserve state when staging
* objects.tree: skip hash computation in digest() when we have already
loaded hash from state
* always check file existence in referencehashfile
* update repofs test since stage no longer allows nonexistent files
* odb: only protect on check() if hash was verified
* staging: use unique memfs namespace for staging per dest ODB
* tests: clean full staging namespace
* refodb: preserve repofs cache dir
* repodependency: stage imports to local ODB and strict check circular
imports
* fetch: fetch staging last
* staging: make dest odb mandatory and preserve state
* refodb: cache nonlocal fs's
* fix rebase errors
* repofs: support full instantiation from fs config
* repofs: preserve subrepo factory for erepos
* tests: clarify repofs dirty file behavior
* objects: migrate remote status to objects.status
* data_cloud: use objects.status
* objects: migrate remote._process to objects.transfer
* data_cloud: use objects.transfer
* objects: handle memfs staging objects in transfer
* fetch/import: use unified transfer()
* update get_remote and transfer exception usage
* update tests
* objects.transfer: use src ODB verify rules after xfer
* odb: move index from remote into odb
* objects.transfer: skip src status check when possible
* update tests
* remove dvc.remote
* odb.index: migrate to diskcache
* objects: fix status/transfer optimization
* update index usage
* state: remove dead sqlite related code
* update cloud tests
* drop unnecessary remove in tests
* use abstract base class in odb index
* status: load trees from local cache instead of remote odb when possible
* handle status when dir cache is explicitly removed
* Migrate to ossfs
fix 6177
* Add verification to object stored on oss
* Change the oss remote's dependency.
pass all of the tests
* Performance optimaztion and ossfs version update
* Remove the argument to set the resumable get/put
1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default
* Use a new api for performance optimization
* Without filesize progress bar version
* With progress bar version
* Bump and add comments
1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger
* Change the name of kwrags following parent class.
* Bump to new ossfs and rewrite upload_fobj
Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj
* Optimization on rm_files
* Some optimization follow the old version.
* Update dvc/fs/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/objects/db/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* bump into ossfs 2021.7.2
* Bump into ossfs 2021.7.3OC
* Switch to real_oss_server
* Switch to a real oss server
* Use the oss bucket located in US.
* Remove some emulator related code
* Add envs to ci
* remove some oss pytest XFAIL
Because we didn't use emulator, we can remove some pytest XFAILs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* fs: use fsspecwrapper in memfs
* odb: add memfs staging ODB in ODBManager
* odb: add read_only attribute for odb instances
* objects: handle memfs paths when staging trees
* odb: remove git ODB
* output: use staging ODB
* RepoDependency: use staging odb for imports
* dvcfs: handle case where dir cache is unavailable
* fix circular import check
* odb: use temp local ODB for staging
* objects: use staging ODB in stage() and tree load
* objects: add typing for add/check/load/stage
* output/dep: use new staging
* add unit tests for local odb staging
* localfs: account for string paths in move()
* include local in odb.by_scheme
* tests: count dir cache like any other object in test_pull_git_imports
* odb: clear staging on gc()
* objects: add tree.get/filter/insert
* output: use obj.filter/get for granular dir commit
* squash filter logic output
* dependency: don't filter imports at the dep level
* don't init staging ODB for external cache
* use staging ODB when computing tree hash for external outs
* remove unneeded force in test granular commit
* use unique-per-manager memfs URLs for staging
* odb: make staging per-ODBManager
* objects: move state dependent staging into ODBManager
* objects: support checking multiple odbs
- staging dependent checks can be done via ODBManager.check
* checkout: do state lookup before explicit stage() in _changed()
* output: use odbmanager based staging
* RepoDependency: use odbmanager based staging
* update obj staging unit tests
* checkout: remove check(), handle individual file exceptions on link()
* diff: use odbmanager based staging
* objects: add ObjectPermissionError for read/write perm errors
* odb: move staging out of odbmanager
* objects: handle staging ODB in objects.stage()
* objects.tree: don't stage anything in tree methods
* odb: clear staged objects on gc()
* remote: don't filter objs by scheme, objs are already grouped by ODB
* diff: update objects.stage usage
* output/dep: update stage() usage
* update ODB tests
* update func tests
* remove memfs from FS_MAP
* revert unneeded objects.check() changes
* remove unused tree.insert()
* revert unused odbmanager changes
* add separate odb error classes
* stage external outs directly in ODB
* check for obj existence in main ODB before staging
* use .dvc/tmp/staging instead of per-odb staging
* gc: only gc loal staging once
* objects: add ExternalRepoFile object type
* RepoDependency: implement get_obj (use ExternalRepoFile)
* drop distinction between used external and used objects
* update tests
* remove unneeded fetch from ExternalRepoFile
* wrap erepo errors as an object error
* update cloud/remote functions for new used_objs behavior
* fix update() rev/locked behavior
* update import tests
* remove ExternalRepoFile
* move get_obj and get_used_objs implementations back into repo dependency
* return objects plus associated ODB (remote) in get_used_objs
* odb: add tmp_dir field and support creating Remote from an odb
* return dict mapping odb -> objects in get_used_objs
* use dummy git odb for git imports
* fetch: support pulling from specific odb
* update cloud funcs for get_used_objs usage
* update tests for used_objs behavior
* use fetch in dep.download
* catch circular imports from local fs repos
* add tests for chained and circular imports
* move tmp_dir into odb config
* update return docstring for repo.used_objs
* objects: make HashInfo & HashFile hashable
* stage/output: collect objects (HashFiles) instead of NamedCache
* remote/data_cloud: use objs instead of NamedCache
* repo: use objects instead of NamedCache
* rerun black after rebase
* remove NamedCache usage from run-cache
* drop NamedCache
* update HashInfo.__hash__
* update unit tests
* revert remote_name change
* start cleaning up test failures
* fix tree obj expansion
* update missing cache logging
* fix external dict handling
* update naive odb.gc
* update state save behavior for pull
* update test_data_cloud
* update test_remote
* fix gc with multiple repos
* objects.tree: support filtering w/preserved hash_info
* fix stage cache restore
* drop fs.path_info from HashFile.__hash__
* objects: add optional name attribute
* revert missing cache logging name/suffix changes
* revert name changes in remote tests
* review fixes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update dvc/remote/base.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
we just need to mark them with needs_internet marker.
Unfortunately, it's not possible to do that automatically
with the fixtures as fixtures are per-session and we need
something on more higher level (which the marker is not passed
through in the pytest).
* objects: merge transfer() with stage()
* objects: drop streamfile
* objects: preserve the signature of get_file_hash
* objects: override hash_jobs if jobs= passed
* objects: support no progress bar directive
* tests: spy on _get_file_hash
* objects: update=... directive
* update: only transfer missing caches with --to-remote
* tests: remove local_remote from the primary test (due to it moves)
* objects: add move=True/False option
* objects: implement _get_file_obj on top of get_file_hash
* objects: use threading on save()
* objects: pass jobs= on to-cache as well
* objects: use in place tqdm
* objects: check for FileExistsError
* Update dvc/objects/__init__.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: get rid of DirInfo
DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.
As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.
* objects: move Tree into tree.py
* Add support for WebHDFS
* Style and linting fixes
* Implement abstract methods and disable no-member pylint errors
* Add init unit test
* Yield client and fix redefinition of hdfs
* Get dynamic hdfs Docker port
* Add support for open in WebHDFSTree
* Remove hdfscli_config fallbacks in WebHDFSTree
* Use port 50070 for webhdfs
* WebHDFSTree supports http URL and proper walking
* Fix failing unit test after refactoring
* Add overwrite argument to upload
* Add remaining func tests for webhdfs
* Remove print statements
* Allow dynamic attributes in pylint for hdfs library
* Comment error on instantiating HDFS client from config
* Remove depth argument from call to walk
* Update dvc/tree/webhdfs.py
* Remove unnecessary deque
* Check exception messages hdfs client failure from config
* Return client directly
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Split long line
* Add size to HashInfo
* Implement move via hdfs.rename
* Add missing line
* Make parent directories before moving file in WebHDFS
* Add total to progress bar
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* state: add simple reentrance
* state: pass Repo instead of LocalCache
* state: use granularly
Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
* dvc: get rid of is_walking_tree
No longer needed. Some applications of `isinstance()` will be refactored
in the future, as trees will just provide the needed functionality.
* dvc: move GitTree to dvc/tree/
* dvc: rename *RemoteTree to *Tree
Usually we need it to parametrize tests with, so there is no
reason to duplicate the code. I've left `local_remote` fixture
just because it is so convenient and often used in a lot of tests.
Daniele Trifirò
Saugat Pachhai (सौगात)
Ruslan Kuprieiev
Batuhan Taskaya
Batuhan Taskaya
Tomás Farías SantanaandRuslan Kuprieiev
nik123