The tests should be in dvc-objects repo npw, and will be migrated soon.
The nanotime and diskcache are specified in dvc-objects, so we don't need
those here.
By splitting dvc.cli into a module, we'll be able to share cli-related
utilities, split parser-related stuffs into smaller parts, improve
reusability and improve API for command extensions in the future.
Initial simple separation to get the process of extracting dvc-objects
and dvc-data packages out of dvc.
dvc-objects is for:
* defining base Object (aka HashFile) and ObjectDB
* providing tools to work with those objects (check/transfer/etc)
* etc
dvc-data is for:
* defining higher level objects (e.g. Tree)
* bulding objects from the workspace (aka staging)
* projecting objects back into the workspace (aka checkout)
* comparing Tree objects (aka diff)
* defining misc object databases (e.g LocalObjectDB)
* higher-level transfer (e.g. with Tree indexes)
* etc
For example, the objects we use in run-cache are based on dvc-objects
and not on dvc-data, as they have absolutely nothing to do with data
management.
There are still some mixups in the code that will be addressed in the
followups.
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.
This PR also removes some redundant/broken remote tests.
Intermediate PR in removing path_info from our code base.
It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.
For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).
Fixes#6774
* objects: migrate remote status to objects.status
* data_cloud: use objects.status
* objects: migrate remote._process to objects.transfer
* data_cloud: use objects.transfer
* objects: handle memfs staging objects in transfer
* fetch/import: use unified transfer()
* update get_remote and transfer exception usage
* update tests
* objects.transfer: use src ODB verify rules after xfer
* odb: move index from remote into odb
* objects.transfer: skip src status check when possible
* update tests
* remove dvc.remote
* odb.index: migrate to diskcache
* objects: fix status/transfer optimization
* update index usage
* state: remove dead sqlite related code
* update cloud tests
* drop unnecessary remove in tests
* use abstract base class in odb index
* status: load trees from local cache instead of remote odb when possible
* handle status when dir cache is explicitly removed
* try to close dvc.state on exit
I have a feeling that it is causing the issues with
shutil.copytree and some PermissionError during 'init',
as we are not closing it between multiple 'main()' calls.
But I am not quite sure, and this seems like a good thing to
do anyway.
* fixup
* close dvc on exit
* close cache on tests and on init
* dvc: get rid of DirInfo
DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.
As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.
* objects: move Tree into tree.py
* Fix partial file checkouts in the same directory
* unittest => pytest
* Try to rationalize changed() to take filter_info
* Use path.read_text() to verify the directory structure
* Try to use specialized hash, fall back to general one if doesn't exist
* Cover subdirectories in hash specialization
* coverage: add a test for empty subdirectories
* Move out the dir_info filtering block
* hash_origin => path_info
* cleanup
* Move hashing functionality after the existence checks
* Simplify it a bit
* Update dvc/cache/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions from code review
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions
* simplify tests
* change filenames
* Some visual changes for consistency
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* #4578 added is_user_executable to dvc file
* Addressed PR feedback and introduced test_add_executable
Also fixed all tests in test_add.py
* Added a test to checkout executable file
* Pop isexec in dvc/output/__init__.py:loadd_from
* Implement test that file is executable after checkout
* Address PR feedback
* Address PR feedback
* Changed BaseTree.isexec to use path_info as an argument
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* fix formatting
* add: preserve exec
* checkout: fix setting exec
* tests: refactor checkout exec test
* tests: fix windows cases
* tests: add: fix exec test on windows
* Update dvc/tree/local.py
* Update dvc/output/base.py
* Update dvc/output/base.py
* Update dvc/cache/base.py
* Update tests/func/test_add.py
* tests: add: skip whole exec test on windows
* save isexec to lockfiles
* fix formatting
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* s3: use mocked s3 instead of the real one
Not removing the fixtures just yet
* Update moto
Tests was failing, because of a change in `responses` package, which
`moto` was mocking. This was fixed on
https://github.com/spulec/moto/pull/3466 which is only available on
`dev122` release.
* Use mocked s3 in checkout tests
Currently we are converting dir_info to/from lists all the time.
The reason is that dir_info is stored as list of dicts in *.dir files,
but that makes it hard to work with. In addition to that, we will likely
be changing .dir file format in the near future #829, so we need to
abstract away dir_info into something that we won't care how it will be
stored on disk.
Related #3256
Related #4847
* Fix error during external outputs checkout
User in discord complained that external outputs' checkout
was failing. Looks like, during refactor of `HashInfo`,
this was forgotten, and as no tests existed before, we had
a regression from 1.6.5 onwards.
* fix tests
* Restyled by black (#4571)
Co-authored-by: Restyled.io <[email protected]>
Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>
Co-authored-by: Restyled.io <[email protected]>
* dvc: get rid of is_walking_tree
No longer needed. Some applications of `isinstance()` will be refactored
in the future, as trees will just provide the needed functionality.
* dvc: move GitTree to dvc/tree/
* dvc: rename *RemoteTree to *Tree
* remote: move constants into trees
* remote: replace Remote() Cache() helper functions
* output: update for remote changes
* cache/data_cloud: update for remote changes
* dependency: update for remote changes
* stage: update for remote changes
* remote: move remaining non-index methods into tree
* tests: update remote unit tests
* remote: LocalCache needs to lock remote index, not cache index
* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH
* tests: repo tree open needs state context
* remote: make Remote.gc a class method
* operates on obj.tree, so that GC can be done with both remote and
cache instances
* bug fixes
* tests: update test_data_cloud
* bug fixes
* tests: update func tests
* remote: move get_file_checksum() into tree
* makes RemoteTree and RepoTree consistent with regard to checksum
calculation
* remote: save() now takes explicit tree parameter
* when tree is remote.tree, save will be a move + link operation
(same as default existing behavior)
* when saving path from a different tree, save will be a copy operation
* tests: update for moved remote/tree functions
* remote: move get_checksum into tree
* remote: separate cloud remote and cache classes
* tests: update unit tests for remote/cache separation
* remote: cloud cache should extend cloud remote
* remote: move LocalRemote.get
* tests: update func tests
* dependency: update for moved get_checksum
* remote: fix state lookup bug
Saugat Pachhai (सौगात)
Ruslan Kuprieiev
Batuhan Taskaya
Christian Clauss
Batuhan Taskaya
