Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-06 15:28:28 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,355 Commits 7 Branches 579 Tags
2.0.0a2
Commit Graph
2142 Commits
Author SHA1 Message Date
Peter Rowlands (변기호) fed6179992 exp run: if target is provided, only checkout target and dependencies (#5519) 
* tests: test that only target and its deps are checked out on exp run

* exp run: if target is provided, only checkout target + deps
 2021-02-25 15:12:27 +09:00
Ruslan Kuprieiev a26f2995ef dir_info: only build trie when needed (#5512) 2021-02-24 09:15:20 +02:00
Ruslan Kuprieiev 502f0eecc9 tests: temporarily skip dvclive tests (#5513) 2021-02-24 08:52:33 +02:00
Ruslan Kuprieiev 8f568ab197 state: use diskcache (#5505) 
* state: only load when needed

* state: don't use separate load/dump

* checkout: group link removal logic together

* state: use diskcache.Cache

* staging: use pbar and save to state

* state: don't pass the whole repo to __init__

* staging: explicitly pass state object

* fs: get rid of odb property

* state: don't close the connection to cache

* repo: close: close state
 2021-02-23 20:51:34 +02:00
Batuhan Taskaya d587d41d6c utils: proper file interface for HashedStreamReader (#5508) 2021-02-23 02:47:36 +02:00
Batuhan Taskaya 174621696d azure: migrate AzureFileSystem to fsspec (#5463) 
* azure: migrate AzureFileSystem to fsspec

* client => fs

* principal auth

* better login error handling
 2021-02-22 12:05:15 +02:00
Batuhan Taskaya e1a193f00b update: forbid updating repo-imports with --to-remote (#5475) 2021-02-22 11:55:00 +02:00
Batuhan Taskaya 8102ede5cc add, import-url: always resolve the output in the local path for --to-remote (#5473) 
* add, import-url: always resolve the output in the local path for --to-remote

* windows

* use the proper fixture
 2021-02-19 15:22:59 +02:00
Saugat Pachhai 14944dd9e0 Minor refactoring stage:list command (#5498) 
Moving text truncation to utils and adding tests for it
 2021-02-19 18:27:37 +05:45
Paweł Redzyński 63f32936b2 live integration: turn on tests (#5487) 2021-02-19 09:25:28 +01:00
Batuhan Taskaya 9ae8bcdf52 gdrive: support service auth with json (#5485) 
* gdrive: support service auth with json

* retry logic
 2021-02-18 17:10:35 +02:00
Peter Rowlands (변기호) 9909ee85a1 git: implement dulwich add --update (#5491) 
* dulwich: implement add with --update

* add git backend test for add()

* fix list comprehension in dulwich status()

* allow adding single string path instead of list

* handle windows paths in add(update=...)
 2021-02-18 17:56:56 +09:00
Saugat Pachhai 45477a2a81 dvc.lock: raise error if the file is gitignored (#5484) 2021-02-18 03:08:00 +02:00
Ruslan Kuprieiev 282e83c481 tests: temporarily exclude gdrivefs from extras (#5483) 
When trying to publish to pypi, we get:

```
Invalid value for requires_dist. Error: Can't have direct dependency:
"gdrivefs @ git+https://github.com/intake/gdrivefs.git ; extra ==
'tests'"
```
 2021-02-17 05:49:34 +02:00
Ruslan Kuprieiev e0842ce01e dvc: rename cache -> odb (#5481) 2021-02-16 22:49:25 +02:00
Ruslan Kuprieiev a814a49181 fs: detach state from fs (#5479) 
We used to think this way, but state is not really an fs concept
(fsspec's dircache is though), as it is a cache of hashes of objects
that we are staging, and the hash doesn't depend on fs, as it is usually
computed by us by reading from fs.

Pre-requisite for diskcache-based state.
 2021-02-16 17:42:50 +02:00
Saugat Pachhai d28f5675b4 remove support for double braces (#5476) 2021-02-16 17:21:47 +02:00
Sami Jawhar 66f24b8708 exp run: Separate exp param parsing from other commands (#5310) 
* Separate exp param parsing from other commands

* Update exp run params metavar

* Updated experiments run param argument name
 2021-02-16 23:19:56 +09:00
Ruslan Kuprieiev 53ba51b5fe objects: separate staging (#5470) 2021-02-16 02:21:54 +02:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
Ruslan Kuprieiev 9ce6bce9b6 tree: move get_hash to object id logic (#5460) 2021-02-13 04:22:29 +02:00
Batuhan TaskayaandRuslan Kuprieiev f4b14c2d4a gdrive: Implement .ls() protocol / proper tree walking with .walk_files() (#5433) 
* gdrive: .ls()/.walk()/.walk_dir()

* better walking

* proper tests

* ._ls_recursive()

* use posxipath for tests

* store ids instead of raw names

* add gdrive fixture

* add dependency

* tests: temporarily disable gdrive tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2021-02-13 03:10:38 +02:00
Ruslan Kuprieiev 7c5e0f2949 tree: get_hash: accept explicit hash name (#5452) 2021-02-11 22:44:28 +02:00
Ruslan Kuprieiev 02cfe33cf9 output: handle missing hash_info.nfiles (#5448) 2021-02-11 14:53:40 +02:00
Ruslan Kuprieiev c635493104 tests: don't limit the number of pytest workers (#5441) 
We used to do this because of gitpython leaks.
 2021-02-11 12:27:00 +02:00
Ruslan Kuprieiev 039c2421d9 cache: misc cleanups (#5447) 
* cache: remove duplicated use_state

* tree: move hash_to* helper to cache
 2021-02-11 11:28:24 +02:00
Peter Rowlands (변기호) e3ee960e8e exp run: remove exp reset in favor of exp run --reset (#5442) 
* exp run: remove `exp reset` in favor of `exp run --reset`

* update command unit test
 2021-02-11 11:53:20 +09:00
Peter Rowlands (변기호) 9d82739b3a exp: add exp remove command (#5443) 
* exp: add remove function for removing specific experiment(s)

* add `dvc exp remove` command

* update tests
 2021-02-11 11:45:25 +09:00
Ruslan Kuprieiev fa550f91c3 cache: move unrelated helpers out of CloudCache class (#5439) 
* cache: move transfer to objects

Moving it as is for now.

* cache: move link() to checkout
 2021-02-10 07:49:27 +02:00
Batuhan Taskaya 384a4113ca update: support --jobs even without --to-remote (#5434) 2021-02-09 17:58:59 +02:00
Ruslan Kuprieiev 0fdb119ed5 cache: move save() to objects (#5430) 2021-02-09 04:56:45 +02:00
Ruslan Kuprieiev 226c68f8ff dvc: introduce File/Tree cache objects (#5424) 2021-02-09 04:00:06 +02:00
Saugat Pachhai 02a0d1af52 stage: implement add/create methods in stage loader (#5397) 
* stage: add/create methods in stage loader

* minor fixups
 2021-02-08 18:30:10 +05:45
Paweł Redzyński a7da6c2a80 live: support cache and no_cache use case (#5394) 
* live: add cache functionality

* live: add tests for cache use case
 2021-02-07 03:51:18 +02:00
Ruslan Kuprieiev b75aab4d85 utils: don't use unnecessary exists() calls (#5423) 
* utils: file_md5: only return hex

* utils: file_md5: don't use exists()

* checkout: don't call exists()
 2021-02-06 16:13:58 +02:00
Pavel Ditenbir e7686f90a6 term: replace 'DVC-file' and 'stage file' with correct terms (#3960) (#5280) 2021-02-05 16:01:42 +09:00
RogerandPeter Rowlands e97f8fe19c pygit2, gitpython: handle relpaths in reset/checkout_index backends (#5414) 
* Make minor change that fixes things for some reason

While working on vscode-dvc, both mattseddon and I found that this change allows
us to run `dvc exp run`, and we cannot otherwise.

I don't exactly know what this change does, and I'm hoping someone more
knowledgable does- after which we can give this PR a better name.

* test relpaths in git backend funcs which take path lists

* gitpython: handle relpaths in reset and checkout_index

* use / for git relpaths on windows

Co-authored-by: Peter Rowlands <[email protected]>
 2021-02-05 12:05:51 +09:00
Ruslan Kuprieiev 320715a5f8 dvc: move checkout out of cache (#5415) 
Checkout is an operation that builds object from cache(aka object
database) in a particular tree(aka filesystem). Thus it shouldn't be
part of the object database itself nor of the filesystem.

Pre-requisite for #5337
 2021-02-05 03:27:00 +02:00
Ruslan Kuprieiev 8676617ae0 cache: separate save() and checkout() (#5412) 2021-02-05 01:54:46 +02:00
Batuhan Taskaya e7cc7765ac config: Support ~/.aws/config parsing (#5378) 
* config: Support ~/.aws/config parsing

* skip when the ~/.aws/config doesn't exist

* Expand windows paths properly
 2021-02-04 14:58:42 +02:00
Saugat Pachhai 21200c14df refactor webdav to use client; extract write_text to base (#5411) 2021-02-04 16:48:25 +05:45
Batuhan Taskaya ce554b6ae7 tree: Introduce the new .ls() API (#5275) 
* s3: faster directory hashing

* Introduce Tree.info() API

* tests for s3

* Implement support for azure

* New .ls() API

* remove _iter_hashes from other trees

* assert recursive

* yield strings instead of file_infos

* Add GSTree.ls()

* Add DETAIL_FIELDS to the GSTree

* WebDAVTree.ls()

* proper webdav root directory initalization

* disable arguments-differ
 2021-02-04 12:10:42 +02:00
Peter Rowlands (변기호) 982802effc experiments: bug fixes (#5407) 
* exp run: display metrics on -m/--metrics

* exp show: fix sort handling
 2021-02-04 16:42:22 +09:00
Ruslan Kuprieiev 985aef883e cli: config: rename --repo to --project (#5399) 
Per https://github.com/iterative/dvc.org/pull/2064

Not changing the internal terminology because we stick to Repo in a lot
of places there and adjusting all of it would be wasteful. Maybe in the
future.
 2021-02-03 17:13:53 +02:00
Batuhan Taskaya fc77618ad1 update: implement --to-remote flag (#5343) 
* upload: implement --to-remote flag

* unit tests

* preserve frozen= for stage
 2021-02-03 12:31:16 +02:00
Saugat Pachhai b2c2929869 pyarrow support for Python3.9 (#5391) 2021-02-02 17:31:11 +02:00
Batuhan Taskaya a92bfd272d get, get-url: add --jobs option (#5362) 2021-02-02 15:49:58 +02:00
Paweł Redzyński 31e4a3325d live integration: rename report to html (#5366) 2021-02-02 13:44:38 +01:00
Saugat Pachhai 67980de44d tests: mock global config dir for Updater (#5386) 2021-02-02 17:08:09 +05:45
Ruslan Kuprieiev 8932e0be84 dvc: get rid of can_be_skipped logic (#5385) 2021-02-02 13:13:48 +02:00