Commit Graph
717 Commits
Author SHA1 Message Date
Paweł Redzyński eccc320606 remote: use configured jobs number (#5747) 2021-04-08 17:52:42 +03:00
Peter Rowlands (변기호) a984f95cbd exp run: jobs should default to 1 (#5785) 2021-04-08 19:27:48 +09:00
Saugat Pachhai 0206812d1b Fix test assertion with environment when using pyenv (#5765) 2021-04-05 22:56:05 +05:45
Saugat PachhaiandSaugat Pachhai 19d0b2af56 use sys.executable instead of hardcoded python (#5763)
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2021-04-03 07:22:07 +00:00
c3a61ad9cf Added SSL verification option (#5732)
* Added SSL verification option

* Changed param name to avoid conflicts

* verify_ssl now accepts strings

* Added simple unit tests

* Changed parameter name fron verify_ssl to ssl_verify so it's consistent with HTTP remotes

Forget to change 1 test

* add processing on ssl_verify param

typo

* revert back to bool param

* Update dvc/config_schema.py

Co-authored-by: Nigel <[email protected]>
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-03-31 23:31:38 +03:00
Saugat Pachhai 7c7e73e3d2 status: collect stages only once (#5741)
This is achieved by using `lock_repo`
context manager that can be used in a recursive depth
that only resets stages when it has left any lock_repo
blocks.

Also added some ui related tests for the status which I found
missing
2021-03-31 18:31:27 +03:00
Saugat Pachhai b5f31daab0 ui: Use sys.stdout and sys.stderr dynamically (#5729)
Before, we used to set it during initialization. But this
made it hard to test using capsys as capsys changes the
reference to sys.stdout and sys.stderr with the mocked one
during the test. But since we are using dvc.ui globally and
setting it to a real sys.stdout and sys.stderr before capsys
comes into effect, testing was not possible with it.

Now, by using property hack, we will be using the values of
sys.stdout and sys.stderr during the time of write, so the mocking
would work and would be easier to test.
2021-03-30 23:44:46 +03:00
dependabot[bot]andSaugat Pachhai 7b922d0ec3 build(deps): bump pytest-mock from 1.11.2 to 3.5.1 (#5705)
* build(deps): bump pytest-mock from 1.11.2 to 3.5.1

Bumps [pytest-mock](https://github.com/pytest-dev/pytest-mock) from 1.11.2 to 3.5.1.
- [Release notes](https://github.com/pytest-dev/pytest-mock/releases)
- [Changelog](https://github.com/pytest-dev/pytest-mock/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest-mock/compare/v1.11.2...v3.5.1)

Signed-off-by: dependabot[bot] <[email protected]>

* change spy.return_value to spy.spy_return

This was introduced in pytest-mock 2.0.0
https://github.com/pytest-dev/pytest-mock/commit/78499644cb800f8fe65a5c17e81d168fa2a4c6b9

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2021-03-26 21:19:37 +05:45
Saugat Pachhai 76ece8c64a plots: quote and resolve paths properly (#5670)
This commit fixes 3 things:
1. The path printed on the terminal is properly is quoted.  This was
problematic when the output is piped to `xdg-open` if the path contained
spaces or quotes or special characters.

2. The path is resolved. This is not an issue at all, but makes the
output nicer.

3. The path with which the browser is opened with uses relative path.
This is done to solve issue in WSL, that the absolute path is not
accessible outside the WSL for host programs as the linux filesystem
is available on `/wsl$` path.
2021-03-24 20:27:24 +05:45
Alex Ford b3bfa346c3 Using existing logging level if defined in addLoggingLevel (#5632)
* Use existing level if defined in addLoggingLevel

Fixes https://github.com/iterative/dvc/issues/4546

* Add addLoggingLevel test

* Format fix.
2021-03-22 15:15:16 +02:00
Peter Rowlands (변기호) 686898240c Revert "repro/exp run: use git status when reminding to git-track files (#5544)" (#5596)
This reverts commit ce2d7c65db.
2021-03-12 11:50:13 +09:00
Saugat Pachhai d245008987 plots: support opening plots file directly in the browser (#5584)
* plots: support opening plots file directly in the browser

* add tests

* fix path on windows tests
2021-03-12 04:23:07 +02:00
Ruslan Kuprieiev 29ad87ebe2 objects: split into separate files (#5609) 2021-03-12 04:12:46 +02:00
Ruslan Kuprieiev 4af3b3d20d dvc: refactor objects (#5607)
* dvc: get rid of DirInfo

DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.

As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.

* objects: move Tree into tree.py
2021-03-12 01:10:34 +02:00
Saugat Pachhai fd66ef6bc1 http: fix info() if there is no content-length header (#5602) 2021-03-11 16:01:24 +02:00
Peter Rowlands (변기호) d68f6d3696 pygit/dulwich: implement stash functionality (#5286)
* pygit2: disable set_ref with reflog message

* git: implement is_dirty in dulwich

* pygit2: implement _stash_apply

* pygit: update to 1.5.0

* enable set_ref(message=...) for pygit2

* git: replace backend reflog_delete in favor of _stash_drop

* pygit: support stash_push with custom refs

* update tests for backend stash_drop behavior

* pygit: raise MergeConflictError on failed stash apply

* pygit: reload index from disk before stash apply

* update stash_apply checkout strategy

* mark flaky test
2021-03-08 17:24:35 +02:00
Ruslan Kuprieiev 06aad3e2b3 dependency: repo: try to fetch before staging (#5549)
Related to https://github.com/iterative/dvc-bench/issues/234
2021-03-05 01:03:33 +02:00
Peter Rowlands (변기호) ce2d7c65db repro/exp run: use git status when reminding to git-track files (#5544)
* repro: skip DVC deps in SCM track stage

(DVC deps will be tracked when their own stage is repro'd)

* scm: use git status when reminding to git-track files

* sort names in reminder instead of random set order

* update unit test
2021-03-04 17:48:12 +09:00
Peter Rowlands (변기호) 1542025afa exp: bash tab completion (#5532)
* exp list: add --names-only option

* completion: add exp name completion

* dvc exp: use exp name completion

* exp: fix bash completion
2021-03-03 15:46:11 +09:00
Batuhan Taskaya 3ebccda62c gs: use etag for the checksum parameter (#5534)
Switch from `md5` to `etag` as the checksum parameter. Due to `etag`
is not preserved while doing server side copies, this patch drops support
for external gs outputs.
2021-03-02 12:43:27 +02:00
Batuhan Taskaya b65dfe04f9 gs: Migrate Google Storage to FSSpec (#5526)
* azure: generalize fsspec functionality into FSSpecWrapper

* gs: migrate google storage to fsspec

* tests: disable empty_dir tests

* setup: add gcsfs to requirements

* gs: decode md5Hash into md5

* gs: speed up downloads via increasing chunk size
2021-03-01 17:15:31 +02:00
Ruslan Kuprieiev a26f2995ef dir_info: only build trie when needed (#5512) 2021-02-24 09:15:20 +02:00
Ruslan Kuprieiev 8f568ab197 state: use diskcache (#5505)
* state: only load when needed

* state: don't use separate load/dump

* checkout: group link removal logic together

* state: use diskcache.Cache

* staging: use pbar and save to state

* state: don't pass the whole repo to __init__

* staging: explicitly pass state object

* fs: get rid of odb property

* state: don't close the connection to cache

* repo: close: close state
2021-02-23 20:51:34 +02:00
Batuhan Taskaya d587d41d6c utils: proper file interface for HashedStreamReader (#5508) 2021-02-23 02:47:36 +02:00
Batuhan Taskaya 174621696d azure: migrate AzureFileSystem to fsspec (#5463)
* azure: migrate AzureFileSystem to fsspec

* client => fs

* principal auth

* better login error handling
2021-02-22 12:05:15 +02:00
Saugat Pachhai 14944dd9e0 Minor refactoring stage:list command (#5498)
Moving text truncation to utils and adding tests for it
2021-02-19 18:27:37 +05:45
Batuhan Taskaya 9ae8bcdf52 gdrive: support service auth with json (#5485)
* gdrive: support service auth with json

* retry logic
2021-02-18 17:10:35 +02:00
Peter Rowlands (변기호) 9909ee85a1 git: implement dulwich add --update (#5491)
* dulwich: implement add with --update

* add git backend test for add()

* fix list comprehension in dulwich status()

* allow adding single string path instead of list

* handle windows paths in add(update=...)
2021-02-18 17:56:56 +09:00
Saugat Pachhai 45477a2a81 dvc.lock: raise error if the file is gitignored (#5484) 2021-02-18 03:08:00 +02:00
Ruslan Kuprieiev e0842ce01e dvc: rename cache -> odb (#5481) 2021-02-16 22:49:25 +02:00
Saugat Pachhai d28f5675b4 remove support for double braces (#5476) 2021-02-16 17:21:47 +02:00
Ruslan Kuprieiev 53ba51b5fe objects: separate staging (#5470) 2021-02-16 02:21:54 +02:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
Ruslan Kuprieiev 9ce6bce9b6 tree: move get_hash to object id logic (#5460) 2021-02-13 04:22:29 +02:00
Batuhan TaskayaandRuslan Kuprieiev f4b14c2d4a gdrive: Implement .ls() protocol / proper tree walking with .walk_files() (#5433)
* gdrive: .ls()/.walk()/.walk_dir()

* better walking

* proper tests

* ._ls_recursive()

* use posxipath for tests

* store ids instead of raw names

* add gdrive fixture

* add dependency

* tests: temporarily disable gdrive tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-02-13 03:10:38 +02:00
Ruslan Kuprieiev 7c5e0f2949 tree: get_hash: accept explicit hash name (#5452) 2021-02-11 22:44:28 +02:00
Peter Rowlands (변기호) e3ee960e8e exp run: remove exp reset in favor of exp run --reset (#5442)
* exp run: remove `exp reset` in favor of `exp run --reset`

* update command unit test
2021-02-11 11:53:20 +09:00
Peter Rowlands (변기호) 9d82739b3a exp: add exp remove command (#5443)
* exp: add remove function for removing specific experiment(s)

* add `dvc exp remove` command

* update tests
2021-02-11 11:45:25 +09:00
Batuhan Taskaya 384a4113ca update: support --jobs even without --to-remote (#5434) 2021-02-09 17:58:59 +02:00
Ruslan Kuprieiev 0fdb119ed5 cache: move save() to objects (#5430) 2021-02-09 04:56:45 +02:00
Ruslan Kuprieiev 226c68f8ff dvc: introduce File/Tree cache objects (#5424) 2021-02-09 04:00:06 +02:00
Saugat Pachhai 02a0d1af52 stage: implement add/create methods in stage loader (#5397)
* stage: add/create methods in stage loader

* minor fixups
2021-02-08 18:30:10 +05:45
Paweł Redzyński a7da6c2a80 live: support cache and no_cache use case (#5394)
* live: add cache functionality

* live: add tests for cache use case
2021-02-07 03:51:18 +02:00
Ruslan Kuprieiev b75aab4d85 utils: don't use unnecessary exists() calls (#5423)
* utils: file_md5: only return hex

* utils: file_md5: don't use exists()

* checkout: don't call exists()
2021-02-06 16:13:58 +02:00
RogerandPeter Rowlands e97f8fe19c pygit2, gitpython: handle relpaths in reset/checkout_index backends (#5414)
* Make minor change that fixes things for some reason

While working on vscode-dvc, both mattseddon and I found that this change allows
us to run `dvc exp run`, and we cannot otherwise.

I don't exactly know what this change does, and I'm hoping someone more
knowledgable does- after which we can give this PR a better name.

* test relpaths in git backend funcs which take path lists

* gitpython: handle relpaths in reset and checkout_index

* use / for git relpaths on windows

Co-authored-by: Peter Rowlands <[email protected]>
2021-02-05 12:05:51 +09:00
Ruslan Kuprieiev 320715a5f8 dvc: move checkout out of cache (#5415)
Checkout is an operation that builds object from cache(aka object
database) in a particular tree(aka filesystem). Thus it shouldn't be
part of the object database itself nor of the filesystem.

Pre-requisite for #5337
2021-02-05 03:27:00 +02:00
Ruslan Kuprieiev 8676617ae0 cache: separate save() and checkout() (#5412) 2021-02-05 01:54:46 +02:00
Batuhan Taskaya e7cc7765ac config: Support ~/.aws/config parsing (#5378)
* config: Support ~/.aws/config parsing

* skip when the ~/.aws/config doesn't exist

* Expand windows paths properly
2021-02-04 14:58:42 +02:00
Batuhan Taskaya fc77618ad1 update: implement --to-remote flag (#5343)
* upload: implement --to-remote flag

* unit tests

* preserve frozen= for stage
2021-02-03 12:31:16 +02:00
Saugat Pachhai b2c2929869 pyarrow support for Python3.9 (#5391) 2021-02-02 17:31:11 +02:00