Commit Graph
3378 Commits
Author SHA1 Message Date
Batuhan Taskaya ec82c3fff5 tests: azure directory generation (#5328) 2021-01-25 15:41:29 +02:00
Saugat Pachhai 9f0b86f221 unit.scm.test_git: use pytest instead of unittest (#5325) 2021-01-25 10:23:46 +05:45
Saugat Pachhai 3741f730e0 show git-add hint on stage:add, refactor scm_context (#5290)
* Refactor scm_context, show git-add hint on stage:add

* move tests
2021-01-24 22:43:38 +02:00
Paweł Redzyński 6a9c0c5ce5 live/checkpoints: test continue (#5300)
* live: test resuming

* initial test, not working

* experiments: introduce DVCLIVE_RESUME env variable on resume
2021-01-24 22:06:37 +02:00
Saugat Pachhai 1d8761e0c9 Remove empty file tests/unit/command/test_commit.py (#5316)
When I opened it, I got confused for a second and had to double-check if it was really empty.
The empty file gives a false assurance. We do need to add one, which I'll add later, but for now,
we better remove it.
2021-01-23 18:27:43 +05:45
Paweł Redzyński 2277a1a655 metrics/plots/params: return ordered (#5297)
* metrics/plots/params: return ordered

* metrics/params/plots: add tests for return order
2021-01-22 20:57:43 +02:00
Ruslan Kuprieiev cbd7472079 dvc: refactor cache saving (#5314) 2021-01-22 19:07:36 +02:00
Ruslan Kuprieiev c4888cbd7d dvc: get rid of ExternalRepo (#5309) 2021-01-21 23:03:53 +02:00
Timofey Yefimovandkarajan1001 60d83a74d7 dvcignore: Fix incorrect ignored output computation (#4986)
* dvcignore: Fix incorrect ignored output computation

* dvcignore: Fixed unignored patterns

Unignored patterns were shown in the dvc status output and caused Error.

Also made a bit of a refactoring in `matches` and `ignore`.

* dvcignore: Fix functional tests

Not sure if I follow, but tests assume that unignored files should trigger `check-ignore`. What was the intention?

* dvcignore: Fix stage.wdir being MagicMock

* Use `is_ignored` instead of `check_ignore`

1. use is_ignored to judge DVC ignore status.
2. use check_ignore to show ignore patterns.

* Solve tests fails

* Add a new tests

* Return to the old check_ignore logic.

* Pass the pylint

* Simplify ignore tests

Co-authored-by: karajan1001 <[email protected]>
2021-01-21 14:28:08 +02:00
Batuhan Taskaya b5068a9200 checkout: don't filter hash's if it is a file (#5298) 2021-01-20 12:36:47 +02:00
Ruslan Kuprieiev 03fc531a5c pylint: enable no-else-return (#5294)
From https://github.com/iterative/dvc/pull/5270#discussion_r559440586
2021-01-19 22:12:46 +02:00
Ruslan Kuprieiev d038bca05e tests: add basic [web]hdfs mockers (#5279) 2021-01-19 20:03:16 +02:00
Batuhan Taskaya 51ada6d272 Implement straight-to-remote functionality (#5198)
* Implement export-to-remote functionality

* Redistrubition of functionality from tree to remote

* Pass chunk size into the open_url

* Move functionality away from remote to cache

* Basic progress bars for both transfer types

* Proper job calculation

* Redesign .open() usage in order to work with different trees

* Align the signatures of other open() functions to accept **kwargs

* Implement chunked uploading to the ssh

* Use local cache to compute directory hash (md5)

* S3: multipart upload test

* Wrap local cache's hash_to_path with PathInfo

* Use .isdir() instead of a try/except statement for calculating the from_infos

* New CLI layout (add/import-url)

* Adjust CLI tests to accept new arguments

* tests: import_url/add func/unit tests

* Introduce cache.transfer()

* Invalid CLI combination tests

* Store dir_info key in the transfer_directory

* LocalTree.upload_fobj

* Tests for utils

* Lazy imports on some places

* --to-remote migration (and repo._transfer)

* Use proper temporary location, more efficient pbar calc

* Directory transfer logic rewrite

* Use shutil.copyfileobj()

* Rebase...

* Better tests

* Inline transfer operation to .add() / .transfer()

* Open in the 'rb' mode for the tests

* More tests, better invalid option handling at repo level

* .read() certain amount of bytes to validate the content

* Better invalid argument handling

* Better progress bars by using .getsize() API

* For fallback upload, use the from_info.name instead of temporary naem

* introduce --jobs for add/import-url
2021-01-19 12:53:32 +02:00
Peter Rowlands (변기호) 407f5607a2 scm: implement pygit2 resolve rev and native checkout (#5270)
* pygit2: implement get/set/remove refs

* pygit: implement iter_refs

* pygit: implement resolve_rev

* pygit2: implement working tree checkout

* pygit2: implement reset, explicitly read/write index when used

* pygit2: checkout does not need to reset index

* pygit2: handle "-" branch shorthand in checkout

* tests: install post-checkout hook should be run via gitpython

* exp apply: use stricter exception handling for merge conflict

* pygit: fix index read/write usage

* gitpython: handle untracked file conflicts in stash apply

* fix flaky/race index issue
2021-01-18 17:41:25 +02:00
Saugat Pachhai 78e12327bc use repo.stage.create_from_cli to create stages (#5281) 2021-01-18 18:55:19 +05:45
Ruslan Kuprieiev 4f2de7a3ef tests: remove psutil checks (#5283)
Leftover from https://github.com/iterative/dvc/pull/5232
2021-01-18 13:50:21 +02:00
Saugat Pachhai 628f87845c introduce stage:add command (#5263)
* Implement stage:add

* fix tests

* Add short flag -C for checkpoints
2021-01-18 10:05:36 +02:00
Ruslan Kuprieiev b027760e12 dvc: move protect from tree to cache (#5271) 2021-01-14 21:10:53 +02:00
Ruslan Kuprieiev a2cac8b423 tests: upgrade azurite and enable loose mode (#5272)
* tests: upgrade azurite and enable loose mode

Azure tests started failing recently. They still work with real azure, so this is on azurite side.

* tests: use cloud instead of workspace

also gdrive doesn't support it yet :(
2021-01-14 20:07:24 +02:00
Ruslan Kuprieiev 3e42e015d6 dvc: redefine is_callback as cmd and no outs/deps (#5187) 2021-01-14 17:23:20 +02:00
Saugat Pachhai 257f58450a refactor run; move to stage.utils (#5262)
* refactor run; move to stage.utils

* Update dvc/repo/run.py

* fix pylint

* skip on run-cache
2021-01-13 21:19:36 +02:00
Batuhan Taskaya c840cecc91 tree: implement getsize() (#5261)
* Implement Tree.getsize() API

* Remove azure from tests

* strip support for recursive size calculation

* _getsize() -> getsize()
2021-01-13 18:35:57 +02:00
Batuhan Taskaya e5f91f5c7f cache: don't initalize cloud cache until its needed (#5264) 2021-01-13 17:01:06 +02:00
Saugat Pachhai f8200ca2b8 move parsing params to utils (#5259)
* move parsing params to utils

* fix tests

* move chunk_dict and test it

* update test
2021-01-13 13:36:49 +02:00
Peter Rowlands (변기호) 1ab1abf62a git: implement merge/checkout via pygit2 (#5245)
* git: implement merge (gitpython, pygit2)

* pylint: add pygit2 as C extension

* git: s/checkout_paths/checkout_index/ and add pygit implementation

* git: support basic checkout/merge conflict strategies in checkout_index

* fix relpath issue in pygit merge

* git: add stubs for add --update

* experiments: drop remaining gitpython specific scm usage
2021-01-12 13:26:27 +02:00
Saugat Pachhai a2c91d2058 scm: implement gitignore (#5243)
* scm: implement gitignore; use `pygit2` backend

* Add is_ignored implementation on `pygit2` and `gitpython` backends.
* Also implement `._reset()` on `repo.scm` which is most of the changes.

* fix tests

* use _backend_func

* minor fixups
2021-01-11 12:31:26 +02:00
Batuhan TaskayaandRuslan Kuprieiev 7fd191126e S3Tree._list_paths now recognizes the max_items limit (#5239)
* S3Tree._list_paths now recognizes the max_items limit

* Use modern approach + add a test

* refactor tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* lint

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-11 11:54:42 +02:00
Batuhan Taskaya 5374880220 s3: Allow authentication using session token (#5240)
* S3: Allow authentication using session token

* Add a little test
2021-01-11 10:48:57 +02:00
Ruslan Kuprieiev c83a4ea406 erepo: refactor download/fetch logic (#5234)
* erepo: don't use _get_tree_for

* erepo: refactor download/fetch logic

* dvc: unify handling existing output
2021-01-10 10:48:36 +02:00
Dmitry Kropachev 6b05cc5e24 dvc.command.diff: add renamed detection feature (#5160)
fixes #5150
2021-01-09 13:49:03 +09:00
Paweł Redzyński 386d0c463a live: add report switch (#5223) 2021-01-07 13:52:21 +02:00
Saugat Pachhai af2a12a188 Use 'schema' top-level element in the lockfile for version (#5222) 2021-01-06 20:08:14 +02:00
mbiesekandRuslan Kuprieiev 00461bb5df commit: support granularity (#4297) (#5088)
* commit: support granularity (#4297)

* commit: support granularity

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-06 19:39:04 +02:00
Batuhan TaskayaandRuslan Kuprieiev 6fb84f641c checkout: fix partial file checkouts in the same directory (#5163)
* Fix partial file checkouts in the same directory

* unittest => pytest

* Try to rationalize changed() to take filter_info

* Use path.read_text() to verify the directory structure

* Try to use specialized hash, fall back to general one if doesn't exist

* Cover subdirectories in hash specialization

* coverage: add a test for empty subdirectories

* Move out the dir_info filtering block

* hash_origin => path_info

* cleanup

* Move hashing functionality after the existence checks

* Simplify it a bit

* Update dvc/cache/base.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Apply suggestions from code review

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Apply suggestions

* simplify tests

* change filenames

* Some visual changes for consistency

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-06 14:21:00 +02:00
Saugat Pachhai 0cc5227f47 Add versioning scheme in the dvc.lock (#5128)
* Add versioning scheme in the dvc.lock

* remove whitespace

* use actual size in tests assumptions
2021-01-06 15:44:19 +05:45
Peter Rowlands (변기호) 37f890444a diff: account for deleted workspace data (#5205)
* diff: add test for deleted data in addition to .dvc file

* diff: handle deleted data when checking for missing cache
2021-01-05 17:52:24 +09:00
Ruslan Kuprieiev b779a117ab add: handle long filenames (#5201) 2021-01-05 10:33:07 +02:00
Paweł RedzyńskiandRuslan Kuprieiev 862ab8de62 live logs initial (#4902)
* plots: allow dir plots

* first scenario

* separate dvclive tests

* rename index to step

* another test

* refactor

* its ok to have step in data

* sort plots

* introduce --logs

* add logs to run commands

* variable name refactoring

* fix logs option

* fixup

* cmd: run: fix unit tests

* plots: modify: fix tests

* improve dir handling

* fixup

* dvclive: logs summary

* api: introduce logs summarization method

* handle no repo case

* fix granular dir plots

* api: dvclive: simplify

* dvclive: roll back logs flag

* cmd: introduce logs

* api: adjust to logs

* test: dvclive fix

* rename logs to dvclive

* live: command: add uninitialized

* outs: dvclive type

* dvclive: support providing config via evn variables

* dvclive: treat as metrics

* dvclive integration: add support for summary on/off

* dvc integration: add temporary test for configuration export

* remove requirements

* cmd: introduce logs

* rename logs to dvclive

* cmd: introduce logs

* remove logs leftowvers

* dvclive: env: adjust to chekpoints env variables

* command: live: remove unnecessary help

* fix linter suggestions

* type hinting

* review refactor

* replace dvclive with live

* stage: loading: move live loading to separate method

* stage: serialize: allow only single live output

* live: add show/diff

* stage: serialize: roll back single live enforcing

* fixup

* rebase fixups

* output: live: force single instance

* live: command: add unit tests

* Update dvc/output/base.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/stage/__init__.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/stage/serialize.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* review v2

* live: fix summary tracking by experiments

* naming fixups

* api: live: add tests

* tests: api: separate live tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-04 11:29:41 +02:00
Ruslan Kuprieiev 7df53e5e26 dvc: handle multiline command as list (#5194)
Implements an approach proposed by @skshetry.

Fixes #4373
2021-01-04 09:11:57 +02:00
Michael StegemanandOLOLO ALALA 764960fbfc cli: config: show merged config with --show-origin (#5188)
* cli: config: show merged config with --show-origin

Builds on #5126 and #5184 by showing the full merged config when
listing or getting config options.

References #5126
References #5184
References iterative/dvc.org#2028
Fixes #5119

* Add additional tests.

* Add test for merged config.

* config: rename methods

Co-authored-by: OLOLO ALALA <[email protected]>
2021-01-04 06:43:31 +02:00
Ruslan Kuprieiev 948633782e run-cache: skip unhashable stages (#5185)
Fixes #5182
2020-12-31 06:28:07 +02:00
Ruslan Kuprieiev ceaba68f62 config: temporarily forbid --show-origin without level (#5184) 2020-12-31 00:54:52 +02:00
Ruslan Kuprieiev 8162bb159c config: read merged config by default (#5179)
* config: read merged config by default

* tests: move root test out of config tests

* tests: refactor config tests

* cli: fix cache dir command
2020-12-31 00:37:39 +02:00
Paweł Redzyński c28e671f47 tests: plots: fix plots call (#5183) 2020-12-30 22:59:32 +02:00
Michael StegemanandRuslan Kuprieiev 34ed545d64 cli: config: add support for --show-origin (#5126)
* cli: config: add support for --show-origin

Support the --show-origin option for config, which, similar to git,
prefixes each config option with the source file it originated from.

Fixes #5119

* Reformat with black.

* Simplify logic for local config files.

* Add test for --show-config output.

* Update tests/func/test_config.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-12-30 21:56:20 +02:00
Pavel Ditenbir 23811e028c cmd: implement '--show-md' flag for 'dvc metrics show' (#4770) (#5161) 2020-12-30 02:17:41 +02:00
Ruslan Kuprieiev 6fcafc9121 erepo: make repo configuration explicit (#5174) 2020-12-29 14:32:29 +02:00
Peter Rowlands (변기호) db555ec66b experiments: fix checkout error for unexisting outputs (#5175)
* stage: support explicit allow_missing in checkout

* exp: explicitly clear EXEC_BRANCH and checkout with allow_missing on
starting experiment run

* git: wrap stash apply errors with SCMError

* git: add `git status` to backends

* exp: reset staged git changes before workspace exp runs to avoid merge
conflicts

* test untracked (and staged) pipeline file and deps for both workspace
and tmpdir runs

* experiments: explicitly track pipeline files and uncached deps for exp
commmits

* fix exp typing errors
2020-12-29 18:07:21 +09:00
Dmitry Kropachev c64b4ce78f dvc/utils/stream: make IterStream seekable and peekable (#5084)
fixes #5021
2020-12-28 23:34:24 +02:00
Saugat Pachhai 01a4473c20 parametrization: fix error when the file in vars is a directory (#5171)
* Fix error when the file in `vars` is a directory

* Fix on windows

* mock tree only
2020-12-28 23:03:09 +02:00