2149 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 06aad3e2b3 dependency: repo: try to fetch before staging (#5549)
Related to https://github.com/iterative/dvc-bench/issues/234
2021-03-05 01:03:33 +02:00
Peter Rowlands (변기호) ce2d7c65db repro/exp run: use git status when reminding to git-track files (#5544)
* repro: skip DVC deps in SCM track stage

(DVC deps will be tracked when their own stage is repro'd)

* scm: use git status when reminding to git-track files

* sort names in reminder instead of random set order

* update unit test
2021-03-04 17:48:12 +09:00
Peter Rowlands (변기호) 1542025afa exp: bash tab completion (#5532)
* exp list: add --names-only option

* completion: add exp name completion

* dvc exp: use exp name completion

* exp: fix bash completion
2021-03-03 15:46:11 +09:00
Paweł Redzyński 4a8cb806d2 live integration: get rid of dvc from live (#5466)
* live integration: get rid of dvc from live

* live: remove from api, add test for html generation during run

* stage: run: convert monitors to class context managers

* run: make monitors run in single thread

* run: move monitor logic out of run
2021-03-03 00:45:35 +09:00
Batuhan Taskaya 3ebccda62c gs: use etag for the checksum parameter (#5534)
Switch from `md5` to `etag` as the checksum parameter. Due to `etag`
is not preserved while doing server side copies, this patch drops support
for external gs outputs.
2021-03-02 12:43:27 +02:00
Batuhan Taskaya b65dfe04f9 gs: Migrate Google Storage to FSSpec (#5526)
* azure: generalize fsspec functionality into FSSpecWrapper

* gs: migrate google storage to fsspec

* tests: disable empty_dir tests

* setup: add gcsfs to requirements

* gs: decode md5Hash into md5

* gs: speed up downloads via increasing chunk size
2021-03-01 17:15:31 +02:00
Paweł Redzyński 92b95bae9b plots: ignore binary plot files (#5523) 2021-03-01 15:00:42 +02:00
Peter Rowlands (변기호) fed6179992 exp run: if target is provided, only checkout target and dependencies (#5519)
* tests: test that only target and its deps are checked out on exp run

* exp run: if target is provided, only checkout target + deps
2021-02-25 15:12:27 +09:00
Ruslan Kuprieiev a26f2995ef dir_info: only build trie when needed (#5512) 2021-02-24 09:15:20 +02:00
Ruslan Kuprieiev 502f0eecc9 tests: temporarily skip dvclive tests (#5513) 2021-02-24 08:52:33 +02:00
Ruslan Kuprieiev 8f568ab197 state: use diskcache (#5505)
* state: only load when needed

* state: don't use separate load/dump

* checkout: group link removal logic together

* state: use diskcache.Cache

* staging: use pbar and save to state

* state: don't pass the whole repo to __init__

* staging: explicitly pass state object

* fs: get rid of odb property

* state: don't close the connection to cache

* repo: close: close state
2021-02-23 20:51:34 +02:00
Batuhan Taskaya d587d41d6c utils: proper file interface for HashedStreamReader (#5508) 2021-02-23 02:47:36 +02:00
Batuhan Taskaya 174621696d azure: migrate AzureFileSystem to fsspec (#5463)
* azure: migrate AzureFileSystem to fsspec

* client => fs

* principal auth

* better login error handling
2021-02-22 12:05:15 +02:00
Batuhan Taskaya e1a193f00b update: forbid updating repo-imports with --to-remote (#5475) 2021-02-22 11:55:00 +02:00
Batuhan Taskaya 8102ede5cc add, import-url: always resolve the output in the local path for --to-remote (#5473)
* add, import-url: always resolve the output in the local path for --to-remote

* windows

* use the proper fixture
2021-02-19 15:22:59 +02:00
Saugat Pachhai 14944dd9e0 Minor refactoring stage:list command (#5498)
Moving text truncation to utils and adding tests for it
2021-02-19 18:27:37 +05:45
Paweł Redzyński 63f32936b2 live integration: turn on tests (#5487) 2021-02-19 09:25:28 +01:00
Batuhan Taskaya 9ae8bcdf52 gdrive: support service auth with json (#5485)
* gdrive: support service auth with json

* retry logic
2021-02-18 17:10:35 +02:00
Peter Rowlands (변기호) 9909ee85a1 git: implement dulwich add --update (#5491)
* dulwich: implement add with --update

* add git backend test for add()

* fix list comprehension in dulwich status()

* allow adding single string path instead of list

* handle windows paths in add(update=...)
2021-02-18 17:56:56 +09:00
Saugat Pachhai 45477a2a81 dvc.lock: raise error if the file is gitignored (#5484) 2021-02-18 03:08:00 +02:00
Ruslan Kuprieiev 282e83c481 tests: temporarily exclude gdrivefs from extras (#5483)
When trying to publish to pypi, we get:

```
Invalid value for requires_dist. Error: Can't have direct dependency:
"gdrivefs @ git+https://github.com/intake/gdrivefs.git ; extra ==
'tests'"
```
2021-02-17 05:49:34 +02:00
Ruslan Kuprieiev e0842ce01e dvc: rename cache -> odb (#5481) 2021-02-16 22:49:25 +02:00
Ruslan Kuprieiev a814a49181 fs: detach state from fs (#5479)
We used to think this way, but state is not really an fs concept
(fsspec's dircache is though), as it is a cache of hashes of objects
that we are staging, and the hash doesn't depend on fs, as it is usually
computed by us by reading from fs.

Pre-requisite for diskcache-based state.
2021-02-16 17:42:50 +02:00
Saugat Pachhai d28f5675b4 remove support for double braces (#5476) 2021-02-16 17:21:47 +02:00
Sami Jawhar 66f24b8708 exp run: Separate exp param parsing from other commands (#5310)
* Separate exp param parsing from other commands

* Update exp run params metavar

* Updated experiments run param argument name
2021-02-16 23:19:56 +09:00
Ruslan Kuprieiev 53ba51b5fe objects: separate staging (#5470) 2021-02-16 02:21:54 +02:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
Ruslan Kuprieiev 9ce6bce9b6 tree: move get_hash to object id logic (#5460) 2021-02-13 04:22:29 +02:00
Batuhan TaskayaandRuslan Kuprieiev f4b14c2d4a gdrive: Implement .ls() protocol / proper tree walking with .walk_files() (#5433)
* gdrive: .ls()/.walk()/.walk_dir()

* better walking

* proper tests

* ._ls_recursive()

* use posxipath for tests

* store ids instead of raw names

* add gdrive fixture

* add dependency

* tests: temporarily disable gdrive tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-02-13 03:10:38 +02:00
Ruslan Kuprieiev 7c5e0f2949 tree: get_hash: accept explicit hash name (#5452) 2021-02-11 22:44:28 +02:00
Ruslan Kuprieiev 02cfe33cf9 output: handle missing hash_info.nfiles (#5448) 2021-02-11 14:53:40 +02:00
Ruslan Kuprieiev c635493104 tests: don't limit the number of pytest workers (#5441)
We used to do this because of gitpython leaks.
2021-02-11 12:27:00 +02:00
Ruslan Kuprieiev 039c2421d9 cache: misc cleanups (#5447)
* cache: remove duplicated use_state

* tree: move hash_to* helper to cache
2021-02-11 11:28:24 +02:00
Peter Rowlands (변기호) e3ee960e8e exp run: remove exp reset in favor of exp run --reset (#5442)
* exp run: remove `exp reset` in favor of `exp run --reset`

* update command unit test
2021-02-11 11:53:20 +09:00
Peter Rowlands (변기호) 9d82739b3a exp: add exp remove command (#5443)
* exp: add remove function for removing specific experiment(s)

* add `dvc exp remove` command

* update tests
2021-02-11 11:45:25 +09:00
Ruslan Kuprieiev fa550f91c3 cache: move unrelated helpers out of CloudCache class (#5439)
* cache: move transfer to objects

Moving it as is for now.

* cache: move link() to checkout
2021-02-10 07:49:27 +02:00
Batuhan Taskaya 384a4113ca update: support --jobs even without --to-remote (#5434) 2021-02-09 17:58:59 +02:00
Ruslan Kuprieiev 0fdb119ed5 cache: move save() to objects (#5430) 2021-02-09 04:56:45 +02:00
Ruslan Kuprieiev 226c68f8ff dvc: introduce File/Tree cache objects (#5424) 2021-02-09 04:00:06 +02:00
Saugat Pachhai 02a0d1af52 stage: implement add/create methods in stage loader (#5397)
* stage: add/create methods in stage loader

* minor fixups
2021-02-08 18:30:10 +05:45
Paweł Redzyński a7da6c2a80 live: support cache and no_cache use case (#5394)
* live: add cache functionality

* live: add tests for cache use case
2021-02-07 03:51:18 +02:00
Ruslan Kuprieiev b75aab4d85 utils: don't use unnecessary exists() calls (#5423)
* utils: file_md5: only return hex

* utils: file_md5: don't use exists()

* checkout: don't call exists()
2021-02-06 16:13:58 +02:00
Pavel Ditenbir e7686f90a6 term: replace 'DVC-file' and 'stage file' with correct terms (#3960) (#5280) 2021-02-05 16:01:42 +09:00
RogerandPeter Rowlands e97f8fe19c pygit2, gitpython: handle relpaths in reset/checkout_index backends (#5414)
* Make minor change that fixes things for some reason

While working on vscode-dvc, both mattseddon and I found that this change allows
us to run `dvc exp run`, and we cannot otherwise.

I don't exactly know what this change does, and I'm hoping someone more
knowledgable does- after which we can give this PR a better name.

* test relpaths in git backend funcs which take path lists

* gitpython: handle relpaths in reset and checkout_index

* use / for git relpaths on windows

Co-authored-by: Peter Rowlands <[email protected]>
2021-02-05 12:05:51 +09:00
Ruslan Kuprieiev 320715a5f8 dvc: move checkout out of cache (#5415)
Checkout is an operation that builds object from cache(aka object
database) in a particular tree(aka filesystem). Thus it shouldn't be
part of the object database itself nor of the filesystem.

Pre-requisite for #5337
2021-02-05 03:27:00 +02:00
Ruslan Kuprieiev 8676617ae0 cache: separate save() and checkout() (#5412) 2021-02-05 01:54:46 +02:00
Batuhan Taskaya e7cc7765ac config: Support ~/.aws/config parsing (#5378)
* config: Support ~/.aws/config parsing

* skip when the ~/.aws/config doesn't exist

* Expand windows paths properly
2021-02-04 14:58:42 +02:00
Saugat Pachhai 21200c14df refactor webdav to use client; extract write_text to base (#5411) 2021-02-04 16:48:25 +05:45
Batuhan Taskaya ce554b6ae7 tree: Introduce the new .ls() API (#5275)
* s3: faster directory hashing

* Introduce Tree.info() API

* tests for s3

* Implement support for azure

* New .ls() API

* remove _iter_hashes from other trees

* assert recursive

* yield strings instead of file_infos

* Add GSTree.ls()

* Add DETAIL_FIELDS to the GSTree

* WebDAVTree.ls()

* proper webdav root directory initalization

* disable arguments-differ
2021-02-04 12:10:42 +02:00
Peter Rowlands (변기호) 982802effc experiments: bug fixes (#5407)
* exp run: display metrics on -m/--metrics

* exp show: fix sort handling
2021-02-04 16:42:22 +09:00