Commit Graph
49 Commits
Author SHA1 Message Date
7275843c08 remote: use .dir checksum existence to infer file contents existence (#3632)
* repo: separate dir cache and file cache in memory

- `used_cache()`/`get_used_cache()` in repo/stage/output now return
  tuples of (dir_cache, file_cache) instead of one flat/merged cache

* update tests for new get_used_cache behavior

* remote: if .dir checksum exists on remote, assume contents also exists

- affects all commands which use `cache_exists()` (remote status)

* push: only upload .dir file after full file contents has been uploaded

* gc: always remove .dir checksums first

* functional tests for push/gc

* repo: support nesting dir caches in NamedCache

* remote: NamedCache updates

* Fix tests

* Fix deepsource warnings

* tests: use pytest mocker fixture

* Update dvc/cache.py

Co-Authored-By: Saugat Pachhai <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
2020-04-14 23:07:58 +03:00
Alexander Schepanovski 6ccae711fe dvc: refactor config (#3298)
* dvc: refactor config

Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:

    with config.edit("local") as conf:
        conf["core"]["remote"] = "new-default"

Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict

* test: do not set cache.type for remotes that doesn't support it

* dvc: fix windows absolute paths validation in config

* test: fix windows separate drive test

* dvc: stylistic fixes in config
2020-02-13 19:35:37 +02:00
Jorge Orpinel ec0c39a28e term: review capitalization and backquotes around "dvc"
rel: https://github.com/iterative/dvc/pull/3156#discussion_r369897073
2020-01-22 20:27:01 -06:00
Mr. Outis 326ba21ed0 py3: remove unneeded string casts 2019-12-31 19:43:15 -06:00
Mr. Outis 6d36b85736 py3: remove string compat related code 2019-12-30 23:49:19 -06:00
Mr. Outis 94a0914bba py3: get rid of future imports 2019-12-30 19:35:48 -06:00
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Alexander Schepanovski 86811aac20 imp: dedup and parallelize fetching imported deps (#2654)
* imp: dedup and parallelize external fetches

This prevents same fetches for external artifacts, groups them by source
and fetches in parallel.

* imp: test that fetch works on non workspace deps
2019-10-23 18:02:50 +03:00
Alexander Schepanovski 30341438e0 gc: added --all-commits (#2643)
* dvc: fix branch suffixes in status/push/pull

* gc: add --all-commits flag

* gc: fix gc warning

Say that we always leave things used in the working tree
2019-10-23 18:00:31 +03:00
Alexander Schepanovski 33b0eefec0 dvc: dedup used cache (#2630)
* dvc: dedup used cache

This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts

* dvc: delay checksum names concatenation
2019-10-21 12:12:56 +03:00
Alexander Schepanovski e2eb4ae0c3 perf: delay cache creation
The older code instantiates non-local cache remotes on Repo creation if
any of those are configured. And remote instantiation loads their API
libs.
2019-09-24 09:03:03 +07:00
Ruslan Kuprieiev 9c5fd0c76c dvc: support group cache mode
In this mode, dvc will set appropriate file/dir permissions to support
sharing cache between different users within the same group.

Fixes #2287

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev 8ef9133194 remote: use RemoteConfig class
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-06 16:00:44 +03:00
Ruslan Kuprieiev 9e9e078de3 config: move logic from CLI to API
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-06 14:41:02 +03:00
Mr. Outis d355b1e66e config: allow cross-referenced remotes
Close #1614
2019-04-08 22:10:03 -05:00
Ruslan Kuprieiev ba8d83e9b1 dvc: refactor dvc project class
1) split Project's methods into separate files;
2) rename Project into Repo(similar to gitpython's Repo)

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-13 03:31:23 +02:00
Ruslan Kuprieiev 4896f08ddf dvc: comply with both flake8 and black
flake8 provides some additional features like unused vars check, etc, so
it makes total sense to use it as well.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-04 00:10:36 +02:00
Ruslan Kuprieiev 4069e34130 dvc: use black instead of flake8
Fixes #1546

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-30 03:48:36 +02:00
Mr. Outis 0d46921ec7 py2: add unicode literals
Fix #1512
2019-01-27 20:40:21 -06:00
Ruslan Kuprieiev 6fec8b16cd RFC: conform to pylint
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-16 14:12:19 +02:00
Ruslan Kuprieiev 484f318c27 remote: local: support relative paths
Fixes #1421

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-04 21:14:03 +02:00
Ruslan Kuprieiev 294b259579 cache: support relative paths
Fixes #1284

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-04 16:12:39 +02:00
Mr. Outis 55eadba0ec cache: Cosmetic changes 💅 (#1321) 2018-11-13 13:30:34 -06:00
Ruslan Kuprieiev 8966d6ce47 dvc: introduce protected mode
This is an opt-in mode, that will make data in the workspace
read-only, thus protecting cache from corruption when hardlinks or
symlinks are used. User can use `dvc unprotect file` command, to
replace read-only link to cache with an editable copy. We can use
this for now to decide if we want to go with a `unified workflow`
in the future or if we want to set unprotected `reflink,copy`
by default.

Fixes #799

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-26 16:07:11 +03:00
Ruslan Kuprieiev 0e9774a3f8 RFC: follow flake8 rules
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-25 17:28:58 +03:00
Ruslan Kuprieiev 716c554b93 remote: allow AZURE cache
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-11 22:27:13 +03:00
Ruslan Kuprieiev 0a951f5ab3 dvc: move cloud/ drivers to remote/
Fixes #741

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-06-20 19:42:33 +03:00
Ruslan Kuprieiev ff0d91737b dvc: initial hdfs support
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-06-10 05:32:16 +03:00
Ruslan Kuprieiev 331e3b2bf9 cache: fix cache_dir resolving
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-06-04 22:28:15 +03:00
Ruslan Kuprieiev 7a7c9e7713 dvc: fix import conflicts
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-06-04 22:28:14 +03:00
Ruslan Kuprieiev 1ef21039c4 remote: use Remote() func to determine remote type 2018-06-04 22:28:14 +03:00
Ruslan Kuprieiev 88397ef430 dvc: use Remote concept in existing code
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-06-04 22:28:14 +03:00
Ruslan Kuprieiev 6f4d0cd74a cache: use atomic move to avoid using Lock()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-06-04 22:28:14 +03:00
Ruslan Kuprieiev f03ea7a915 dvc: move link_state from Cache to Project
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-06-04 22:28:14 +03:00
Ruslan Kuprieiev c7fec74516 dvc: store list of links to cleanup later
Fixes #685

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-05-13 21:02:54 +03:00
Ruslan Kuprieiev 21fc712edf cache: remove unused code
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-04-28 21:50:49 +03:00
Ruslan Kuprieiev 0a049c232b cache: configurable cache dir and cache type
From now on we allow specifying standalone cache dirs as well as
specifying linking method(copy, symlink, hardlink, reflink) for it.
If 'cache type' is not specified in the config, dvc will try all of
them starting from reflink all the way down to copy.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-04-28 21:50:49 +03:00
Ruslan Kuprieiev 0cd0e6700f cache: check of cache corruption
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-03-12 00:58:45 +02:00
Ruslan Kuprieiev 54a6532360 cache: create cache dirs only when needed
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-03-08 19:45:58 +02:00
Ruslan Kuprieiev 488b49a852 cache: fix upper limit for sub dirs
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-03-02 15:03:49 +02:00
Ruslan Kuprieiev 539ac41b70 dvc: use two-level hash for cache
Fixes #488

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-03-02 02:51:20 +02:00
Dmitry 5f46f05992 Bugfix: create a cache dir for freshly checkout-ed repo 2018-01-09 18:23:28 -08:00
Ruslan Kuprieiev 1bbfa5ca35 dvc: cleanup artifacts from #382
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-01-07 10:49:38 +02:00
Dmitry 06627e8b85 Custome merge with #379 2018-01-06 18:47:50 -08:00
Dmitry 018302e208 New command: dvc fsck. Close #381. 2018-01-06 19:32:17 +02:00
Dmitry d232a15e8e Refactoring: move find_cache() to Cache class 2018-01-06 18:41:38 +02:00
Dmitry d2790f9306 Refactoring: move find_cache() to Cache class 2018-01-06 04:15:23 -08:00
Ruslan Kuprieiev f47a2c7585 dvc: initial unittests
Initial unittests for new design.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-01-01 16:52:06 +02:00
Ruslan Kuprieiev a779424a36 WIP RFC
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2017-12-23 09:37:40 +02:00