Commit Graph
21 Commits
Author SHA1 Message Date
Peter Rowlands (변기호) 3d84df930b objects: migrate remote push/pull to objects.transfer (#6308)
* objects: migrate remote status to objects.status

* data_cloud: use objects.status

* objects: migrate remote._process to objects.transfer

* data_cloud: use objects.transfer

* objects: handle memfs staging objects in transfer

* fetch/import: use unified transfer()

* update get_remote and transfer exception usage

* update tests

* objects.transfer: use src ODB verify rules after xfer

* odb: move index from remote into odb

* objects.transfer: skip src status check when possible

* update tests

* remove dvc.remote

* odb.index: migrate to diskcache

* objects: fix status/transfer optimization

* update index usage

* state: remove dead sqlite related code

* update cloud tests

* drop unnecessary remove in tests

* use abstract base class in odb index

* status: load trees from local cache instead of remote odb when possible

* handle status when dir cache is explicitly removed
2021-07-20 13:30:01 +03:00
Peter Rowlands (변기호) 71bae7e4af objects: use separate staging ODB for staging trees (#6195)
* fs: use fsspecwrapper in memfs

* odb: add memfs staging ODB in ODBManager

* odb: add read_only attribute for odb instances

* objects: handle memfs paths when staging trees

* odb: remove git ODB

* output: use staging ODB

* RepoDependency: use staging odb for imports

* dvcfs: handle case where dir cache is unavailable

* fix circular import check

* odb: use temp local ODB for staging

* objects: use staging ODB in stage() and tree load

* objects: add typing for add/check/load/stage

* output/dep: use new staging

* add unit tests for local odb staging

* localfs: account for string paths in move()

* include local in odb.by_scheme

* tests: count dir cache like any other object in test_pull_git_imports

* odb: clear staging on gc()

* objects: add tree.get/filter/insert

* output: use obj.filter/get for granular dir commit

* squash filter logic output

* dependency: don't filter imports at the dep level

* don't init staging ODB for external cache

* use staging ODB when computing tree hash for external outs

* remove unneeded force in test granular commit

* use unique-per-manager memfs URLs for staging

* odb: make staging per-ODBManager

* objects: move state dependent staging into ODBManager

* objects: support checking multiple odbs

- staging dependent checks can be done via ODBManager.check

* checkout: do state lookup before explicit stage() in _changed()

* output: use odbmanager based staging

* RepoDependency: use odbmanager based staging

* update obj staging unit tests

* checkout: remove check(), handle individual file exceptions on link()

* diff: use odbmanager based staging

* objects: add ObjectPermissionError for read/write perm errors

* odb: move staging out of odbmanager

* objects: handle staging ODB in objects.stage()

* objects.tree: don't stage anything in tree methods

* odb: clear staged objects on gc()

* remote: don't filter objs by scheme, objs are already grouped by ODB

* diff: update objects.stage usage

* output/dep: update stage() usage

* update ODB tests

* update func tests

* remove memfs from FS_MAP

* revert unneeded objects.check() changes

* remove unused tree.insert()

* revert unused odbmanager changes

* add separate odb error classes

* stage external outs directly in ODB

* check for obj existence in main ODB before staging

* use .dvc/tmp/staging instead of per-odb staging

* gc: only gc loal staging once
2021-07-09 18:48:28 +03:00
Ruslan Kuprieiev 5fb181fcf8 odb: don't use fs.path_info (#6059)
* odb: don't use fs.path_info

* fs: get rid of fs.path_info
2021-06-02 20:43:51 +03:00
Saugat Pachhai a172067c7d fix errors raised by pylint (#6081)
astroid==2.5.7 has improved inference and was throwing
errors that were false negatives before. Fixing these
and ignoring some.

Also added `funcy.decorator` to the `signature-mutators`,
as it changes function signature (i.e. provides `Call` as
a first argument which is not required to pass).

Example errors that we were getting because of this:

```
tests/remotes/gdrive.py:40:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
************* Module dvc.stage.serialize
dvc/stage/serialize.py:38:1: E1120: No value for argument 'func' in
function call (no-value-for-parameter)
************* Module dvc.fs.http
dvc/fs/http.py:19:1: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
************* Module dvc.fs.gdrive
dvc/fs/gdrive.py:66:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
dvc/fs/gdrive.py:459:5: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
```
2021-05-31 05:46:18 +00:00
Saugat Pachhai b9000ed926 pre-commit: update black to 21.5b1 (#6032)
* pre-commit: update black to 21.5b1

also updated others, but black is more noticeable

* remove magic trailing comma
2021-05-18 15:51:15 +00:00
Ruslan Kuprieiev 89b40afee7 fs: make constructor closer to fsspec (#5911)
* fs: don't use repo for jobs/hash_jobs

* fs: local: exists: use fsspec's LocalFileSystem

* fs: make constructor comply with fsspec
2021-05-12 22:03:09 +03:00
b349b39d62 Decoupling dvcignore and fs (#5812)
* Decoupling dvcignore and fs

[x] remove dvcignore from fs objects
[x] attach dvcignore to repo
[x] add ignore check to functions(add, checkout, output, pipeline, etc)
[x] pass all tests
[x] final check

* Solve all of the arugments error.

* add dvcignore to dvc list

* Solve some test_ignore cases

* add dvcignore to checkout.

1. add dvcignore to 'checkout', 'add'
2. add ignore_filter to fs

* Solve two tests

* Get rid of ignore_filter

* Add ignore examine to download

* Solve a test issue

* Solve dvcignore and subrepo problems

* Solve sub repo error

* Revert some changes

* pass all tests

* Remove debug codes

* Some final exams

* dvcignore trie tree only update when needed

* reuse of the dir result in path walk.

* Some change after code review.

* Recover something from false `--force` push

* add abspath trans to dvcginore

* pass pylint

* pylint related

* Update dvc/fs/repo.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Changes related to reviews.

* dvcignore call api change

from dvcignore.walk_files(fs.walk(path_info)) to dvcignore.walk_files(fs, path_info)

* Get rid of PathInfo in test_ignore.py

* fs.walk follow the arguments of os.walk

* Dvcignore as parameters

* pass dvcignore as arguments to checkouts

* Decoupling state and repo

* Update dvc/dvcfile.py

Co-authored-by: Paweł Redzyński <[email protected]>

* Update dvc/objects/stage.py

Co-authored-by: Paweł Redzyński <[email protected]>

* Solve some small problems

* Add tests to verify that dependency now not ignore

fix#5256

* Update dvc/ignore.py

* Update dvc/fs/git.py

* Update dvc/repo/__init__.py

* Some small improvements

* pass black

* Refactor is_ignored

* Update dvc/output/base.py

* repofs: use dvcignore in exists/isdir/isfile

Per https://github.com/iterative/dvc/pull/5812/files#r630604524

Co-authored-by: myname <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-05-12 05:29:04 +03:00
Ruslan Kuprieiev 29ad87ebe2 objects: split into separate files (#5609) 2021-03-12 04:12:46 +02:00
Ruslan Kuprieiev 4af3b3d20d dvc: refactor objects (#5607)
* dvc: get rid of DirInfo

DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.

As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.

* objects: move Tree into tree.py
2021-03-12 01:10:34 +02:00
Ruslan Kuprieiev a26f2995ef dir_info: only build trie when needed (#5512) 2021-02-24 09:15:20 +02:00
Ruslan Kuprieiev 8f568ab197 state: use diskcache (#5505)
* state: only load when needed

* state: don't use separate load/dump

* checkout: group link removal logic together

* state: use diskcache.Cache

* staging: use pbar and save to state

* state: don't pass the whole repo to __init__

* staging: explicitly pass state object

* fs: get rid of odb property

* state: don't close the connection to cache

* repo: close: close state
2021-02-23 20:51:34 +02:00
Ruslan Kuprieiev a814a49181 fs: detach state from fs (#5479)
We used to think this way, but state is not really an fs concept
(fsspec's dircache is though), as it is a cache of hashes of objects
that we are staging, and the hash doesn't depend on fs, as it is usually
computed by us by reading from fs.

Pre-requisite for diskcache-based state.
2021-02-16 17:42:50 +02:00
Ruslan Kuprieiev 53ba51b5fe objects: separate staging (#5470) 2021-02-16 02:21:54 +02:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
Ruslan Kuprieiev 9ce6bce9b6 tree: move get_hash to object id logic (#5460) 2021-02-13 04:22:29 +02:00
Ruslan Kuprieiev 7c5e0f2949 tree: get_hash: accept explicit hash name (#5452) 2021-02-11 22:44:28 +02:00
Ruslan Kuprieiev 039c2421d9 cache: misc cleanups (#5447)
* cache: remove duplicated use_state

* tree: move hash_to* helper to cache
2021-02-11 11:28:24 +02:00
Ruslan Kuprieiev fa550f91c3 cache: move unrelated helpers out of CloudCache class (#5439)
* cache: move transfer to objects

Moving it as is for now.

* cache: move link() to checkout
2021-02-10 07:49:27 +02:00
Ruslan Kuprieiev 226c68f8ff dvc: introduce File/Tree cache objects (#5424) 2021-02-09 04:00:06 +02:00
Ruslan Kuprieiev b75aab4d85 utils: don't use unnecessary exists() calls (#5423)
* utils: file_md5: only return hex

* utils: file_md5: don't use exists()

* checkout: don't call exists()
2021-02-06 16:13:58 +02:00
Ruslan Kuprieiev 320715a5f8 dvc: move checkout out of cache (#5415)
Checkout is an operation that builds object from cache(aka object
database) in a particular tree(aka filesystem). Thus it shouldn't be
part of the object database itself nor of the filesystem.

Pre-requisite for #5337
2021-02-05 03:27:00 +02:00