Commit Graph
2063 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 4f2de7a3ef tests: remove psutil checks (#5283)
Leftover from https://github.com/iterative/dvc/pull/5232
2021-01-18 13:50:21 +02:00
Saugat Pachhai 628f87845c introduce stage:add command (#5263)
* Implement stage:add

* fix tests

* Add short flag -C for checkpoints
2021-01-18 10:05:36 +02:00
Ruslan Kuprieiev b027760e12 dvc: move protect from tree to cache (#5271) 2021-01-14 21:10:53 +02:00
Ruslan Kuprieiev a2cac8b423 tests: upgrade azurite and enable loose mode (#5272)
* tests: upgrade azurite and enable loose mode

Azure tests started failing recently. They still work with real azure, so this is on azurite side.

* tests: use cloud instead of workspace

also gdrive doesn't support it yet :(
2021-01-14 20:07:24 +02:00
Ruslan Kuprieiev 3e42e015d6 dvc: redefine is_callback as cmd and no outs/deps (#5187) 2021-01-14 17:23:20 +02:00
Saugat Pachhai 257f58450a refactor run; move to stage.utils (#5262)
* refactor run; move to stage.utils

* Update dvc/repo/run.py

* fix pylint

* skip on run-cache
2021-01-13 21:19:36 +02:00
Batuhan Taskaya c840cecc91 tree: implement getsize() (#5261)
* Implement Tree.getsize() API

* Remove azure from tests

* strip support for recursive size calculation

* _getsize() -> getsize()
2021-01-13 18:35:57 +02:00
Batuhan Taskaya e5f91f5c7f cache: don't initalize cloud cache until its needed (#5264) 2021-01-13 17:01:06 +02:00
Saugat Pachhai f8200ca2b8 move parsing params to utils (#5259)
* move parsing params to utils

* fix tests

* move chunk_dict and test it

* update test
2021-01-13 13:36:49 +02:00
Peter Rowlands (변기호) 1ab1abf62a git: implement merge/checkout via pygit2 (#5245)
* git: implement merge (gitpython, pygit2)

* pylint: add pygit2 as C extension

* git: s/checkout_paths/checkout_index/ and add pygit implementation

* git: support basic checkout/merge conflict strategies in checkout_index

* fix relpath issue in pygit merge

* git: add stubs for add --update

* experiments: drop remaining gitpython specific scm usage
2021-01-12 13:26:27 +02:00
Saugat Pachhai a2c91d2058 scm: implement gitignore (#5243)
* scm: implement gitignore; use `pygit2` backend

* Add is_ignored implementation on `pygit2` and `gitpython` backends.
* Also implement `._reset()` on `repo.scm` which is most of the changes.

* fix tests

* use _backend_func

* minor fixups
2021-01-11 12:31:26 +02:00
Batuhan TaskayaandRuslan Kuprieiev 7fd191126e S3Tree._list_paths now recognizes the max_items limit (#5239)
* S3Tree._list_paths now recognizes the max_items limit

* Use modern approach + add a test

* refactor tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* lint

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-11 11:54:42 +02:00
Batuhan Taskaya 5374880220 s3: Allow authentication using session token (#5240)
* S3: Allow authentication using session token

* Add a little test
2021-01-11 10:48:57 +02:00
Ruslan Kuprieiev c83a4ea406 erepo: refactor download/fetch logic (#5234)
* erepo: don't use _get_tree_for

* erepo: refactor download/fetch logic

* dvc: unify handling existing output
2021-01-10 10:48:36 +02:00
Dmitry Kropachev 6b05cc5e24 dvc.command.diff: add renamed detection feature (#5160)
fixes #5150
2021-01-09 13:49:03 +09:00
Paweł Redzyński 386d0c463a live: add report switch (#5223) 2021-01-07 13:52:21 +02:00
Saugat Pachhai af2a12a188 Use 'schema' top-level element in the lockfile for version (#5222) 2021-01-06 20:08:14 +02:00
mbiesekandRuslan Kuprieiev 00461bb5df commit: support granularity (#4297) (#5088)
* commit: support granularity (#4297)

* commit: support granularity

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-06 19:39:04 +02:00
Batuhan TaskayaandRuslan Kuprieiev 6fb84f641c checkout: fix partial file checkouts in the same directory (#5163)
* Fix partial file checkouts in the same directory

* unittest => pytest

* Try to rationalize changed() to take filter_info

* Use path.read_text() to verify the directory structure

* Try to use specialized hash, fall back to general one if doesn't exist

* Cover subdirectories in hash specialization

* coverage: add a test for empty subdirectories

* Move out the dir_info filtering block

* hash_origin => path_info

* cleanup

* Move hashing functionality after the existence checks

* Simplify it a bit

* Update dvc/cache/base.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Apply suggestions from code review

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Apply suggestions

* simplify tests

* change filenames

* Some visual changes for consistency

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-06 14:21:00 +02:00
Saugat Pachhai 0cc5227f47 Add versioning scheme in the dvc.lock (#5128)
* Add versioning scheme in the dvc.lock

* remove whitespace

* use actual size in tests assumptions
2021-01-06 15:44:19 +05:45
Peter Rowlands (변기호) 37f890444a diff: account for deleted workspace data (#5205)
* diff: add test for deleted data in addition to .dvc file

* diff: handle deleted data when checking for missing cache
2021-01-05 17:52:24 +09:00
Ruslan Kuprieiev b779a117ab add: handle long filenames (#5201) 2021-01-05 10:33:07 +02:00
Paweł RedzyńskiandRuslan Kuprieiev 862ab8de62 live logs initial (#4902)
* plots: allow dir plots

* first scenario

* separate dvclive tests

* rename index to step

* another test

* refactor

* its ok to have step in data

* sort plots

* introduce --logs

* add logs to run commands

* variable name refactoring

* fix logs option

* fixup

* cmd: run: fix unit tests

* plots: modify: fix tests

* improve dir handling

* fixup

* dvclive: logs summary

* api: introduce logs summarization method

* handle no repo case

* fix granular dir plots

* api: dvclive: simplify

* dvclive: roll back logs flag

* cmd: introduce logs

* api: adjust to logs

* test: dvclive fix

* rename logs to dvclive

* live: command: add uninitialized

* outs: dvclive type

* dvclive: support providing config via evn variables

* dvclive: treat as metrics

* dvclive integration: add support for summary on/off

* dvc integration: add temporary test for configuration export

* remove requirements

* cmd: introduce logs

* rename logs to dvclive

* cmd: introduce logs

* remove logs leftowvers

* dvclive: env: adjust to chekpoints env variables

* command: live: remove unnecessary help

* fix linter suggestions

* type hinting

* review refactor

* replace dvclive with live

* stage: loading: move live loading to separate method

* stage: serialize: allow only single live output

* live: add show/diff

* stage: serialize: roll back single live enforcing

* fixup

* rebase fixups

* output: live: force single instance

* live: command: add unit tests

* Update dvc/output/base.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/stage/__init__.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/stage/serialize.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* review v2

* live: fix summary tracking by experiments

* naming fixups

* api: live: add tests

* tests: api: separate live tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-04 11:29:41 +02:00
Ruslan Kuprieiev 7df53e5e26 dvc: handle multiline command as list (#5194)
Implements an approach proposed by @skshetry.

Fixes #4373
2021-01-04 09:11:57 +02:00
Michael StegemanandOLOLO ALALA 764960fbfc cli: config: show merged config with --show-origin (#5188)
* cli: config: show merged config with --show-origin

Builds on #5126 and #5184 by showing the full merged config when
listing or getting config options.

References #5126
References #5184
References iterative/dvc.org#2028
Fixes #5119

* Add additional tests.

* Add test for merged config.

* config: rename methods

Co-authored-by: OLOLO ALALA <[email protected]>
2021-01-04 06:43:31 +02:00
Ruslan Kuprieiev 948633782e run-cache: skip unhashable stages (#5185)
Fixes #5182
2020-12-31 06:28:07 +02:00
Ruslan Kuprieiev ceaba68f62 config: temporarily forbid --show-origin without level (#5184) 2020-12-31 00:54:52 +02:00
Ruslan Kuprieiev 8162bb159c config: read merged config by default (#5179)
* config: read merged config by default

* tests: move root test out of config tests

* tests: refactor config tests

* cli: fix cache dir command
2020-12-31 00:37:39 +02:00
Paweł Redzyński c28e671f47 tests: plots: fix plots call (#5183) 2020-12-30 22:59:32 +02:00
Michael StegemanandRuslan Kuprieiev 34ed545d64 cli: config: add support for --show-origin (#5126)
* cli: config: add support for --show-origin

Support the --show-origin option for config, which, similar to git,
prefixes each config option with the source file it originated from.

Fixes #5119

* Reformat with black.

* Simplify logic for local config files.

* Add test for --show-config output.

* Update tests/func/test_config.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-12-30 21:56:20 +02:00
Pavel Ditenbir 23811e028c cmd: implement '--show-md' flag for 'dvc metrics show' (#4770) (#5161) 2020-12-30 02:17:41 +02:00
Ruslan Kuprieiev 6fcafc9121 erepo: make repo configuration explicit (#5174) 2020-12-29 14:32:29 +02:00
Peter Rowlands (변기호) db555ec66b experiments: fix checkout error for unexisting outputs (#5175)
* stage: support explicit allow_missing in checkout

* exp: explicitly clear EXEC_BRANCH and checkout with allow_missing on
starting experiment run

* git: wrap stash apply errors with SCMError

* git: add `git status` to backends

* exp: reset staged git changes before workspace exp runs to avoid merge
conflicts

* test untracked (and staged) pipeline file and deps for both workspace
and tmpdir runs

* experiments: explicitly track pipeline files and uncached deps for exp
commmits

* fix exp typing errors
2020-12-29 18:07:21 +09:00
Dmitry Kropachev c64b4ce78f dvc/utils/stream: make IterStream seekable and peekable (#5084)
fixes #5021
2020-12-28 23:34:24 +02:00
Saugat Pachhai 01a4473c20 parametrization: fix error when the file in vars is a directory (#5171)
* Fix error when the file in `vars` is a directory

* Fix on windows

* mock tree only
2020-12-28 23:03:09 +02:00
Peter Rowlands (변기호) 7a92ff5e5d experiments: commit with --no-verify in executors (#5078)
* git: add no_verify flag for commit()

* experiments: commit with --no-verify in executors

* git: wrap hook exceptions with SCMError

* update hook file perms

* update dulwich to 0.20.15

* update pygit2 commit def

* skip hook tests on windows
2020-12-28 17:35:35 +09:00
Ioana Grigoropol fa4bf40eb4 api: add globbing option for pushing (#5033)
* api: add globbing option for pushing

Related: #4816

Signed-off-by: Ioana Grigoropol <[email protected]>

* api: use utility function for push command

Signed-off-by: Ioana Grigoropol <[email protected]>
2020-12-25 23:50:36 +02:00
Dmitry KropachevandRuslan Kuprieiev eb8e03a465 dvc/tree/local: fix it for no repo/cache case (#5118)
* dvc/tree/local: fix it for no repo/cache case

fixes #5069

* Update dvc/output/base.py

* Update dvc/output/base.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-12-25 05:56:04 +02:00
devramx 0bb3f40be8 remove: delete .gitignore file if empty (#5148)
* remove: delete .gitignore file if empty

modified behavior to unlink empty .gitignore files.
added corresponding unit & func test cases.
updated failing test cases.

Fixes #4962

* fix failing lint checks

* integrating feedback from the code review
2020-12-23 23:04:19 +02:00
Ruslan Kuprieiev a6151eaf9e erepo: get rid of get_checksum (#5145) 2020-12-22 15:07:32 +02:00
Paweł Redzyński 4d36bc4de8 params: introduce --targets (#5122)
* params: add targets option

* fixup

* remove unnecessary changes

* fix isort

* params: diff: add test for --targets

* tests: roll back metric/plots unification, introduce collect test

* metrics/plots: refactoring

* missing plots test added
2020-12-22 04:19:13 +02:00
dcea83e75b [WIP] dvc: preserve exec bit for tracked files (#5061)
* #4578 added is_user_executable to dvc file

* Addressed PR feedback and introduced test_add_executable

Also fixed all tests in test_add.py

* Added a test to checkout executable file

* Pop isexec in dvc/output/__init__.py:loadd_from

* Implement test that file is executable after checkout

* Address PR feedback

* Address PR feedback

* Changed BaseTree.isexec to use path_info as an argument

* Update tests/func/test_add.py

* Update tests/func/test_add.py

* Update tests/func/test_add.py

* Update tests/func/test_add.py

* fix formatting

* add: preserve exec

* checkout: fix setting exec

* tests: refactor checkout exec test

* tests: fix windows cases

* tests: add: fix exec test on windows

* Update dvc/tree/local.py

* Update dvc/output/base.py

* Update dvc/output/base.py

* Update dvc/cache/base.py

* Update tests/func/test_add.py

* tests: add: skip whole exec test on windows

* save isexec to lockfiles

* fix formatting

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-12-20 02:31:06 +02:00
Ioana GrigoropolandRuslan Kuprieiev f51bef040a pull: add glob option (#5032)
* api: add glob option for pull command

Related: #4816

Signed-off-by: Ioana Grigoropol <[email protected]>

* api: add globbing utility function

Related: #4816

Signed-off-by: Ioana Grigoropol <[email protected]>

* api: use utility function for pull command

Signed-off-by: Ioana Grigoropol <[email protected]>

* Update dvc/utils/__init__.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-12-19 21:38:40 +02:00
Peter Rowlands (변기호) f5d46c2fdf experiments: run single experiments in user's workspace by default (#5124)
* experiments: move executor into a module

* experiments: separate executor into base/local

* rename LocalExecutor to TempDirExecutor, run in .dvc/tmp by default

* git: restore HEAD using reset not checkout in detached_head()

* experiments: support calling BaseExecutor.reproduce directly for
workspace runs

* experiments: run single experiments in local workspace by default

* update experiments tests for workspace changes

* experiments: update checkpoints for workspace runs

* update checkpoints tests

* update test_show

* add force param for apply

* exp apply: add --force option

* exp run/res: add --temp option to force tempdir execution for single
experiments

* force apply in non-apply specific tests
2020-12-18 18:55:26 +09:00
Ruslan Kuprieiev 64dfaffd7c [WIP] repro: simplify the logic for frozen stages (#5114)
Fixes #5082
2020-12-16 14:28:00 +02:00
Ruslan Kuprieiev aec8d65791 repro: accept multiple targets (#5111)
Currently, we iterate over targets manually in CLI, which results in a
complicated code and a DAG collection/check overhead. With this PR we
collect the DAG once, figure out all the needed targets within it and
just reproduce them in one pass.
2020-12-16 03:25:18 +02:00
Peter Rowlands (변기호) 6b9d842b26 experiments: handle exp run from within subdirs/subrepos (#5093)
* experiments: add test cases for stage in subdir & subrepo

* git: fix relpath handling in dulwich add()

* repro: move exp file git staging from stage.run into repro

* experiments: support running experiments from outside DVC root

* experiments: setup logger inside multiprocessing (executor) context

* git: fix dulwich add in submodules
2020-12-15 19:54:04 +02:00
Sandeep MistryandRuslan Kuprieiev 31502aee0a diff: add support for --targets (#4938)
* wip: add target support (back) to dvc diff

* Change diff param to targets and update repo.diff accordingly

* Update dvc diff --targets with PR review feedback

* Tests for dvc diff --targets

* Update dvc diff --targets with PR review feedback

* Update dvc diff --targets with PR review feedback and additional tests

* split out new diff target tests as per PR comment

* remove unnecessary cache dir removal from setup_targets_test

* Update dvc/command/diff.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-12-15 16:19:26 +02:00
Saugat Pachhai 11c94b10f2 disallow overriding item and key in the foreach loop (#5104)
* Allow reserving keys in the context to prevent overriding

* Use nullcontext from funcy (3.6 Python)

* Display path in error message
2020-12-15 19:06:39 +05:45
Saugat Pachhai 620d082b0b refactor resolver and optimize (#5077)
minor refactors, code comments

Rename interim to definitions

provide location of entry during init for definition

small fixups to set_temporarily

Refactor loop to use dict comprehension

Provide same interface for definitions

Make it work for composite lists

Check for sequence rather than a list

refactor tests

Fixes #4955
Fixes #4957
Fixes #4963
2020-12-14 19:22:15 +05:45