Commit Graph
13 Commits
Author SHA1 Message Date
Batuhan Taskaya 2172add905 fs: introduce tree.find() (#5879) 2021-04-28 13:39:19 +03:00
Batuhan Taskaya a42e363217 objects: don't recompute hash on odb.add() if verify is False (#5860) 2021-04-22 13:50:24 +03:00
Batuhan TaskayaandRuslan Kuprieiev 68897aa0de update: only update the missing hashes with --to-remote (#5773)
* objects: merge transfer() with stage()

* objects: drop streamfile

* objects: preserve the signature of get_file_hash

* objects: override hash_jobs if jobs= passed

* objects: support no progress bar directive

* tests: spy on _get_file_hash

* objects: update=... directive

* update: only transfer missing caches with --to-remote

* tests: remove local_remote from the primary test (due to it moves)

* objects: add move=True/False option

* objects: implement _get_file_obj on top of get_file_hash

* objects: use threading on save()

* objects: pass jobs= on to-cache as well

* objects: use in place tqdm

* objects: check for FileExistsError

* Update dvc/objects/__init__.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-04-19 18:09:35 +03:00
Paweł Redzyński eccc320606 remote: use configured jobs number (#5747) 2021-04-08 17:52:42 +03:00
Ruslan Kuprieiev 2db55032c4 objects: file: remove unused load() method (#5610) 2021-03-12 05:35:48 +02:00
Ruslan Kuprieiev 29ad87ebe2 objects: split into separate files (#5609) 2021-03-12 04:12:46 +02:00
Ruslan Kuprieiev 4af3b3d20d dvc: refactor objects (#5607)
* dvc: get rid of DirInfo

DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.

As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.

* objects: move Tree into tree.py
2021-03-12 01:10:34 +02:00
Ruslan Kuprieiev 06aad3e2b3 dependency: repo: try to fetch before staging (#5549)
Related to https://github.com/iterative/dvc-bench/issues/234
2021-03-05 01:03:33 +02:00
Ruslan Kuprieiev a26f2995ef dir_info: only build trie when needed (#5512) 2021-02-24 09:15:20 +02:00
Ruslan Kuprieiev 8f568ab197 state: use diskcache (#5505)
* state: only load when needed

* state: don't use separate load/dump

* checkout: group link removal logic together

* state: use diskcache.Cache

* staging: use pbar and save to state

* state: don't pass the whole repo to __init__

* staging: explicitly pass state object

* fs: get rid of odb property

* state: don't close the connection to cache

* repo: close: close state
2021-02-23 20:51:34 +02:00
Ruslan Kuprieiev e0842ce01e dvc: rename cache -> odb (#5481) 2021-02-16 22:49:25 +02:00
Ruslan Kuprieiev a814a49181 fs: detach state from fs (#5479)
We used to think this way, but state is not really an fs concept
(fsspec's dircache is though), as it is a cache of hashes of objects
that we are staging, and the hash doesn't depend on fs, as it is usually
computed by us by reading from fs.

Pre-requisite for diskcache-based state.
2021-02-16 17:42:50 +02:00
Ruslan Kuprieiev 53ba51b5fe objects: separate staging (#5470) 2021-02-16 02:21:54 +02:00