Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,316 Commits 7 Branches 579 Tags
f4b14c2d4a462c593ac7b777322e171d2407f1fd
Commit Graph
6316 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Batuhan TaskayaandRuslan Kuprieiev f4b14c2d4a gdrive: Implement .ls() protocol / proper tree walking with .walk_files() (#5433) 
* gdrive: .ls()/.walk()/.walk_dir()

* better walking

* proper tests

* ._ls_recursive()

* use posxipath for tests

* store ids instead of raw names

* add gdrive fixture

* add dependency

* tests: temporarily disable gdrive tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2021-02-13 03:10:38 +02:00
Peter Rowlands (변기호) 273b02fc54 metrics/params: fix bug preventing usage from being displayed (#5453) 2021-02-12 10:46:13 +09:00
Ruslan Kuprieiev 7c5e0f2949 tree: get_hash: accept explicit hash name (#5452) 2021-02-11 22:44:28 +02:00
Saugat Pachhai ac9819eb02 introduce dvc.io module to consolidate io utilities (#5348) 
* add dvc.ui module

The name of class has been changed to `Console` as the short form
name is too bad.

* add demo
 2021-02-11 20:42:08 +02:00
Ruslan Kuprieiev 02cfe33cf9 output: handle missing hash_info.nfiles (#5448) 2021-02-11 14:53:40 +02:00
Ruslan Kuprieiev c635493104 tests: don't limit the number of pytest workers (#5441) 
We used to do this because of gitpython leaks.
 2021-02-11 12:27:00 +02:00
Ruslan Kuprieiev 039c2421d9 cache: misc cleanups (#5447) 
* cache: remove duplicated use_state

* tree: move hash_to* helper to cache
 2021-02-11 11:28:24 +02:00
Peter Rowlands (변기호) e3ee960e8e exp run: remove exp reset in favor of exp run --reset (#5442) 
* exp run: remove `exp reset` in favor of `exp run --reset`

* update command unit test
 2021-02-11 11:53:20 +09:00
Peter Rowlands (변기호) 9d82739b3a exp: add exp remove command (#5443) 
* exp: add remove function for removing specific experiment(s)

* add `dvc exp remove` command

* update tests
 2021-02-11 11:45:25 +09:00
Ruslan Kuprieiev fa550f91c3 cache: move unrelated helpers out of CloudCache class (#5439) 
* cache: move transfer to objects

Moving it as is for now.

* cache: move link() to checkout
 2021-02-10 07:49:27 +02:00
Batuhan Taskaya 384a4113ca update: support --jobs even without --to-remote (#5434) 2021-02-09 17:58:59 +02:00
Peter Rowlands (변기호) b370465209 exp show: customize rich table rendering behavior (#5381) 
* exp show: use rich native pager support

* exp show: re-enable column header styles

* utils: add custom classes to override rich table collapse behavior

* set final collapsed column width to 1 to allow ellipsis
 2021-02-09 23:34:04 +09:00
Peter Rowlands (변기호) 748bda8950 exp run: clean untracked lockfiles before running (#5426) 
* exp run/reset: clean untracked lockfiles

* update log level for run -f/--force message
 2021-02-09 21:38:46 +09:00
Ruslan Kuprieiev 47bdf523d7 cache: add: accept object id and it's path (#5431) 
Unnatural leftover from previous PRs
 2021-02-09 08:15:50 +02:00
Ruslan Kuprieiev 0fdb119ed5 cache: move save() to objects (#5430) 2021-02-09 04:56:45 +02:00
Ruslan Kuprieiev 226c68f8ff dvc: introduce File/Tree cache objects (#5424) 2021-02-09 04:00:06 +02:00
Saugat Pachhai 02a0d1af52 stage: implement add/create methods in stage loader (#5397) 
* stage: add/create methods in stage loader

* minor fixups
 2021-02-08 18:30:10 +05:45
Peter Rowlands (변기호) eeeaa9d324 exp: cleanup executor temp directories ourselves (#5425) 2021-02-08 16:30:24 +09:00
Paweł Redzyński a7da6c2a80 live: support cache and no_cache use case (#5394) 
* live: add cache functionality

* live: add tests for cache use case
 2021-02-07 03:51:18 +02:00
Ruslan Kuprieiev b75aab4d85 utils: don't use unnecessary exists() calls (#5423) 
* utils: file_md5: only return hex

* utils: file_md5: don't use exists()

* checkout: don't call exists()
 2021-02-06 16:13:58 +02:00
Ruslan Kuprieiev f30d51ca42 cache: use add() to add objects to cache (#5421) 
This makes all cache objects go through the same logic, so that all of
the appropriate rules are applied to them (e.g. proper permissions).

Pre-requisite for #5337
 2021-02-06 03:07:29 +02:00
Ruslan Kuprieiev 4b938c5104 gha: temporarily disable macos 11 
Those workers take a very long time to start up.
 2021-02-06 03:07:17 +02:00
Peter Rowlands (변기호) 550cbb5e11 git: update dulwich dependency to 0.20.18 (#5420) 2021-02-05 20:43:44 +09:00
Batuhan Taskaya 5900151ebd gs: fix test failures caused by .ls() (#5419) 2021-02-05 12:09:04 +02:00
Pavel Ditenbir e7686f90a6 term: replace 'DVC-file' and 'stage file' with correct terms (#3960) (#5280) 2021-02-05 16:01:42 +09:00
RogerandPeter Rowlands e97f8fe19c pygit2, gitpython: handle relpaths in reset/checkout_index backends (#5414) 
* Make minor change that fixes things for some reason

While working on vscode-dvc, both mattseddon and I found that this change allows
us to run `dvc exp run`, and we cannot otherwise.

I don't exactly know what this change does, and I'm hoping someone more
knowledgable does- after which we can give this PR a better name.

* test relpaths in git backend funcs which take path lists

* gitpython: handle relpaths in reset and checkout_index

* use / for git relpaths on windows

Co-authored-by: Peter Rowlands <[email protected]>
 2021-02-05 12:05:51 +09:00
Ruslan Kuprieiev 320715a5f8 dvc: move checkout out of cache (#5415) 
Checkout is an operation that builds object from cache(aka object
database) in a particular tree(aka filesystem). Thus it shouldn't be
part of the object database itself nor of the filesystem.

Pre-requisite for #5337
 2021-02-05 03:27:00 +02:00
Ruslan Kuprieiev 8676617ae0 cache: separate save() and checkout() (#5412) 2021-02-05 01:54:46 +02:00
Batuhan Taskaya e7cc7765ac config: Support ~/.aws/config parsing (#5378) 
* config: Support ~/.aws/config parsing

* skip when the ~/.aws/config doesn't exist

* Expand windows paths properly
 2021-02-04 14:58:42 +02:00
Saugat Pachhai 21200c14df refactor webdav to use client; extract write_text to base (#5411) 2021-02-04 16:48:25 +05:45
Batuhan Taskaya ce554b6ae7 tree: Introduce the new .ls() API (#5275) 
* s3: faster directory hashing

* Introduce Tree.info() API

* tests for s3

* Implement support for azure

* New .ls() API

* remove _iter_hashes from other trees

* assert recursive

* yield strings instead of file_infos

* Add GSTree.ls()

* Add DETAIL_FIELDS to the GSTree

* WebDAVTree.ls()

* proper webdav root directory initalization

* disable arguments-differ
 2021-02-04 12:10:42 +02:00
Peter Rowlands (변기호) 79dafe8310 exp gc: prune any exp/exec refs which point to gc'd experiments (#5408) 2021-02-04 17:50:04 +09:00
Peter Rowlands (변기호) 982802effc experiments: bug fixes (#5407) 
* exp run: display metrics on -m/--metrics

* exp show: fix sort handling
 2021-02-04 16:42:22 +09:00
Ruslan Kuprieiev b87e76aa84 tree: stop using self.path_info (#5403) 
* webhdfs: don't use self.path_info

* azure: don't use self.path_info

* webdav: don't use self.path_info

* oss: don't use self.path_info

* http: don't use self.path_info

* gdrive: don't use self.path_info
 2021-02-03 20:50:30 +02:00
Ruslan Kuprieiev 2c81bf1748 api: use Repo.open (#5401) 2021-02-03 18:39:40 +02:00
Batuhan Taskaya adedd19c52 gha: switch to pre-commit/action for caching (#5389) 2021-02-03 17:14:46 +02:00
Ruslan Kuprieiev 985aef883e cli: config: rename --repo to --project (#5399) 
Per https://github.com/iterative/dvc.org/pull/2064

Not changing the internal terminology because we stick to Repo in a lot
of places there and adjusting all of it would be wasteful. Maybe in the
future.
 2021-02-03 17:13:53 +02:00
Ruslan Kuprieiev ffaac0f088 repo: remove unneeded filter (#5393) 
Leftover from #5385
 2021-02-03 17:13:36 +02:00
Batuhan Taskaya fc77618ad1 update: implement --to-remote flag (#5343) 
* upload: implement --to-remote flag

* unit tests

* preserve frozen= for stage
 2021-02-03 12:31:16 +02:00
Saugat Pachhai b2c2929869 pyarrow support for Python3.9 (#5391) 2021-02-02 17:31:11 +02:00
Batuhan Taskaya a92bfd272d get, get-url: add --jobs option (#5362) 2021-02-02 15:49:58 +02:00
Paweł Redzyński 31e4a3325d live integration: rename report to html (#5366) 2021-02-02 13:44:38 +01:00
Saugat Pachhai 67980de44d tests: mock global config dir for Updater (#5386) 2021-02-02 17:08:09 +05:45
Ruslan Kuprieiev 8932e0be84 dvc: get rid of can_be_skipped logic (#5385) 2021-02-02 13:13:48 +02:00
Batuhan Taskaya 9ad1e0e159 ssh: show total-size when uploading/downloading on ssh remote (#5383) 2021-02-02 11:27:19 +02:00
Saugat Pachhai 4348832b44 keep stage.dump and dvcfile.dump in sync (#5382) 2021-02-02 15:11:13 +05:45
Peter Rowlands (변기호) d05be36694 exp show: cleanup UI issues (#5361) 
* exp show: add -n/--num to display the last N commits from HEAD

* exp run: fix empty commit with no changes

* add test for exp show with multiple commits

* allow -n greater than num commits in repo

* review fix
 2021-02-02 11:35:02 +09:00
Peter Rowlands (변기호) 2ea5ef07dd exp apply: force apply by default (#5360) 
* exp apply: force apply by default

* update tests
 2021-02-02 11:07:19 +09:00
Batuhan Taskaya 8205c84077 pull, fetch: optimize memory consuption by ~%35 (#5346) 2021-02-01 18:46:02 +02:00
Saugat Pachhai b020918f5d stage:add: use remainder as a command script (#5350) 
* stage:add use remainder as a command

* use -c for checkpoints

* use -n for stage name, make it required, fix tests
 2021-02-01 17:50:04 +02:00