708 Commits
Author SHA1 Message Date
Alex Ford b3bfa346c3 Using existing logging level if defined in addLoggingLevel (#5632)
* Use existing level if defined in addLoggingLevel

Fixes https://github.com/iterative/dvc/issues/4546

* Add addLoggingLevel test

* Format fix.
2021-03-22 15:15:16 +02:00
Peter Rowlands (변기호) 686898240c Revert "repro/exp run: use git status when reminding to git-track files (#5544)" (#5596)
This reverts commit ce2d7c65db.
2021-03-12 11:50:13 +09:00
Saugat Pachhai d245008987 plots: support opening plots file directly in the browser (#5584)
* plots: support opening plots file directly in the browser

* add tests

* fix path on windows tests
2021-03-12 04:23:07 +02:00
Ruslan Kuprieiev 29ad87ebe2 objects: split into separate files (#5609) 2021-03-12 04:12:46 +02:00
Ruslan Kuprieiev 4af3b3d20d dvc: refactor objects (#5607)
* dvc: get rid of DirInfo

DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.

As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.

* objects: move Tree into tree.py
2021-03-12 01:10:34 +02:00
Saugat Pachhai fd66ef6bc1 http: fix info() if there is no content-length header (#5602) 2021-03-11 16:01:24 +02:00
Peter Rowlands (변기호) d68f6d3696 pygit/dulwich: implement stash functionality (#5286)
* pygit2: disable set_ref with reflog message

* git: implement is_dirty in dulwich

* pygit2: implement _stash_apply

* pygit: update to 1.5.0

* enable set_ref(message=...) for pygit2

* git: replace backend reflog_delete in favor of _stash_drop

* pygit: support stash_push with custom refs

* update tests for backend stash_drop behavior

* pygit: raise MergeConflictError on failed stash apply

* pygit: reload index from disk before stash apply

* update stash_apply checkout strategy

* mark flaky test
2021-03-08 17:24:35 +02:00
Ruslan Kuprieiev 06aad3e2b3 dependency: repo: try to fetch before staging (#5549)
Related to https://github.com/iterative/dvc-bench/issues/234
2021-03-05 01:03:33 +02:00
Peter Rowlands (변기호) ce2d7c65db repro/exp run: use git status when reminding to git-track files (#5544)
* repro: skip DVC deps in SCM track stage

(DVC deps will be tracked when their own stage is repro'd)

* scm: use git status when reminding to git-track files

* sort names in reminder instead of random set order

* update unit test
2021-03-04 17:48:12 +09:00
Peter Rowlands (변기호) 1542025afa exp: bash tab completion (#5532)
* exp list: add --names-only option

* completion: add exp name completion

* dvc exp: use exp name completion

* exp: fix bash completion
2021-03-03 15:46:11 +09:00
Batuhan Taskaya 3ebccda62c gs: use etag for the checksum parameter (#5534)
Switch from `md5` to `etag` as the checksum parameter. Due to `etag`
is not preserved while doing server side copies, this patch drops support
for external gs outputs.
2021-03-02 12:43:27 +02:00
Batuhan Taskaya b65dfe04f9 gs: Migrate Google Storage to FSSpec (#5526)
* azure: generalize fsspec functionality into FSSpecWrapper

* gs: migrate google storage to fsspec

* tests: disable empty_dir tests

* setup: add gcsfs to requirements

* gs: decode md5Hash into md5

* gs: speed up downloads via increasing chunk size
2021-03-01 17:15:31 +02:00
Ruslan Kuprieiev a26f2995ef dir_info: only build trie when needed (#5512) 2021-02-24 09:15:20 +02:00
Ruslan Kuprieiev 8f568ab197 state: use diskcache (#5505)
* state: only load when needed

* state: don't use separate load/dump

* checkout: group link removal logic together

* state: use diskcache.Cache

* staging: use pbar and save to state

* state: don't pass the whole repo to __init__

* staging: explicitly pass state object

* fs: get rid of odb property

* state: don't close the connection to cache

* repo: close: close state
2021-02-23 20:51:34 +02:00
Batuhan Taskaya d587d41d6c utils: proper file interface for HashedStreamReader (#5508) 2021-02-23 02:47:36 +02:00
Batuhan Taskaya 174621696d azure: migrate AzureFileSystem to fsspec (#5463)
* azure: migrate AzureFileSystem to fsspec

* client => fs

* principal auth

* better login error handling
2021-02-22 12:05:15 +02:00
Saugat Pachhai 14944dd9e0 Minor refactoring stage:list command (#5498)
Moving text truncation to utils and adding tests for it
2021-02-19 18:27:37 +05:45
Batuhan Taskaya 9ae8bcdf52 gdrive: support service auth with json (#5485)
* gdrive: support service auth with json

* retry logic
2021-02-18 17:10:35 +02:00
Peter Rowlands (변기호) 9909ee85a1 git: implement dulwich add --update (#5491)
* dulwich: implement add with --update

* add git backend test for add()

* fix list comprehension in dulwich status()

* allow adding single string path instead of list

* handle windows paths in add(update=...)
2021-02-18 17:56:56 +09:00
Saugat Pachhai 45477a2a81 dvc.lock: raise error if the file is gitignored (#5484) 2021-02-18 03:08:00 +02:00
Ruslan Kuprieiev e0842ce01e dvc: rename cache -> odb (#5481) 2021-02-16 22:49:25 +02:00
Saugat Pachhai d28f5675b4 remove support for double braces (#5476) 2021-02-16 17:21:47 +02:00
Ruslan Kuprieiev 53ba51b5fe objects: separate staging (#5470) 2021-02-16 02:21:54 +02:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
Ruslan Kuprieiev 9ce6bce9b6 tree: move get_hash to object id logic (#5460) 2021-02-13 04:22:29 +02:00
Batuhan TaskayaandRuslan Kuprieiev f4b14c2d4a gdrive: Implement .ls() protocol / proper tree walking with .walk_files() (#5433)
* gdrive: .ls()/.walk()/.walk_dir()

* better walking

* proper tests

* ._ls_recursive()

* use posxipath for tests

* store ids instead of raw names

* add gdrive fixture

* add dependency

* tests: temporarily disable gdrive tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-02-13 03:10:38 +02:00
Ruslan Kuprieiev 7c5e0f2949 tree: get_hash: accept explicit hash name (#5452) 2021-02-11 22:44:28 +02:00
Peter Rowlands (변기호) e3ee960e8e exp run: remove exp reset in favor of exp run --reset (#5442)
* exp run: remove `exp reset` in favor of `exp run --reset`

* update command unit test
2021-02-11 11:53:20 +09:00
Peter Rowlands (변기호) 9d82739b3a exp: add exp remove command (#5443)
* exp: add remove function for removing specific experiment(s)

* add `dvc exp remove` command

* update tests
2021-02-11 11:45:25 +09:00
Batuhan Taskaya 384a4113ca update: support --jobs even without --to-remote (#5434) 2021-02-09 17:58:59 +02:00
Ruslan Kuprieiev 0fdb119ed5 cache: move save() to objects (#5430) 2021-02-09 04:56:45 +02:00
Ruslan Kuprieiev 226c68f8ff dvc: introduce File/Tree cache objects (#5424) 2021-02-09 04:00:06 +02:00
Saugat Pachhai 02a0d1af52 stage: implement add/create methods in stage loader (#5397)
* stage: add/create methods in stage loader

* minor fixups
2021-02-08 18:30:10 +05:45
Paweł Redzyński a7da6c2a80 live: support cache and no_cache use case (#5394)
* live: add cache functionality

* live: add tests for cache use case
2021-02-07 03:51:18 +02:00
Ruslan Kuprieiev b75aab4d85 utils: don't use unnecessary exists() calls (#5423)
* utils: file_md5: only return hex

* utils: file_md5: don't use exists()

* checkout: don't call exists()
2021-02-06 16:13:58 +02:00
RogerandPeter Rowlands e97f8fe19c pygit2, gitpython: handle relpaths in reset/checkout_index backends (#5414)
* Make minor change that fixes things for some reason

While working on vscode-dvc, both mattseddon and I found that this change allows
us to run `dvc exp run`, and we cannot otherwise.

I don't exactly know what this change does, and I'm hoping someone more
knowledgable does- after which we can give this PR a better name.

* test relpaths in git backend funcs which take path lists

* gitpython: handle relpaths in reset and checkout_index

* use / for git relpaths on windows

Co-authored-by: Peter Rowlands <[email protected]>
2021-02-05 12:05:51 +09:00
Ruslan Kuprieiev 320715a5f8 dvc: move checkout out of cache (#5415)
Checkout is an operation that builds object from cache(aka object
database) in a particular tree(aka filesystem). Thus it shouldn't be
part of the object database itself nor of the filesystem.

Pre-requisite for #5337
2021-02-05 03:27:00 +02:00
Ruslan Kuprieiev 8676617ae0 cache: separate save() and checkout() (#5412) 2021-02-05 01:54:46 +02:00
Batuhan Taskaya e7cc7765ac config: Support ~/.aws/config parsing (#5378)
* config: Support ~/.aws/config parsing

* skip when the ~/.aws/config doesn't exist

* Expand windows paths properly
2021-02-04 14:58:42 +02:00
Batuhan Taskaya fc77618ad1 update: implement --to-remote flag (#5343)
* upload: implement --to-remote flag

* unit tests

* preserve frozen= for stage
2021-02-03 12:31:16 +02:00
Saugat Pachhai b2c2929869 pyarrow support for Python3.9 (#5391) 2021-02-02 17:31:11 +02:00
Batuhan Taskaya a92bfd272d get, get-url: add --jobs option (#5362) 2021-02-02 15:49:58 +02:00
Paweł Redzyński 31e4a3325d live integration: rename report to html (#5366) 2021-02-02 13:44:38 +01:00
Saugat Pachhai 67980de44d tests: mock global config dir for Updater (#5386) 2021-02-02 17:08:09 +05:45
Saugat Pachhai 4348832b44 keep stage.dump and dvcfile.dump in sync (#5382) 2021-02-02 15:11:13 +05:45
Peter Rowlands (변기호) d05be36694 exp show: cleanup UI issues (#5361)
* exp show: add -n/--num to display the last N commits from HEAD

* exp run: fix empty commit with no changes

* add test for exp show with multiple commits

* allow -n greater than num commits in repo

* review fix
2021-02-02 11:35:02 +09:00
Peter Rowlands (변기호) 2ea5ef07dd exp apply: force apply by default (#5360)
* exp apply: force apply by default

* update tests
2021-02-02 11:07:19 +09:00
Saugat Pachhai b020918f5d stage:add: use remainder as a command script (#5350)
* stage:add use remainder as a command

* use -c for checkpoints

* use -n for stage name, make it required, fix tests
2021-02-01 17:50:04 +02:00
Batuhan Taskaya c84833bab1 params: support unicode characters (#5363)
* params: support unicode characters

* open remove() calls with utf-8
2021-02-01 15:56:53 +02:00
Piotr Bielak 3feaa86696 Pass missing glob flag to pull command (#5354)
The `--glob` flag is parsed from the command line, but in the
`CmdDataPull` class in the invocation the parameter (`self.args.glob`)
is not passed to the `pull` method implementation. Hence, always the
default value (False) is used. This commit fixes this issue and sets the
`glob` parameter properly.
2021-02-01 14:21:21 +02:00