1858 Commits
Author SHA1 Message Date
01abf7c19c Fix error during external outputs checkout (#4570)
* Fix error during external outputs checkout

User in discord complained that external outputs' checkout
was failing. Looks like, during refactor of `HashInfo`,
this was forgotten, and as no tests existed before, we had
a regression from 1.6.5 onwards.

* fix tests

* Restyled by black (#4571)

Co-authored-by: Restyled.io <[email protected]>

Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>
Co-authored-by: Restyled.io <[email protected]>
2020-09-15 11:07:09 +03:00
karajan1001 5e69e29b2f check-ignore show sub-repo patterns (#4564)
* `check-ignore` show sub-repo patterns

fix #4555
1. Sub-repo pattern now would be considered in `check-ignore` command.
2. New outputs for sub-repo patterns in a `detail`
3. Related tests.

* message change
2020-09-15 06:59:41 +03:00
Ruslan Kuprieiev 8c19492ba6 tree: validate resolved remote:// sections (#4569)
* tree: validate resolved remote:// sections

* dvc: use checksum_jobs instead of hash_jobs

This is a regression introduced during one of the refactorings.
2020-09-14 12:15:38 +03:00
Ruslan Kuprieiev 424396b068 tree: resolve deep remote:// urls (#4568)
Fixes #4556
2020-09-14 05:49:54 +03:00
karajan1001andRuslan Kuprieiev 8b16af965c Print dvc version info for debugging (#4536)
* Print dvc version info for debugging

fix #4095
1. seperate info from dvc version.
2. print dvc version info when some unexpected error occured.

* Psutil NoneType error

* Update tests/func/test_version.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Move `package` outside dvc/utils/pkg.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-09-13 04:08:20 +03:00
Ruslan Kuprieiev cf7e534e2d tree: repo: handle dirty state (#4554) 2020-09-11 17:25:33 +03:00
Saugat Pachhai 2931d30ca5 dvc: change dvcfile.dump to do synchronized dump by default (#4544)
The API of `Dvcfile.dump` has been changed to
`dump(stage, update_pipeline=True, update_lock=True)`,
which will make it to always dump by default to both files,
and provide flags to suppress one or the other.

Fixes #4235
2020-09-10 13:12:00 +05:45
Robert F. DickersonandRuslan Kuprieiev 7fa90e425d Allow config option for HTTP method for uploads (Artifactory) (#4553)
* allow for a config option for http method

* add tests

* fix tests with method

* Update dvc/tree/http.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-09-10 05:11:41 +03:00
Victor 54a48e3256 Feature/http ssl verify configuration (#4547)
* add test for http ssl_verify configuration

* add ssl_verify flag to HTTPTree
2020-09-09 19:27:43 +03:00
Ruslan Kuprieiev 1ca2dc19c2 tree: open: handle dirty state in RepoTree (#4548)
We used to use that logic in DvcTree back before we had proper trees.
Now we have a RepoTree, that combines git/workspace tree with the
DvcTree and dirty state clearly belongs to the git/workspace and not
the dvc itself, as dvc state only depends on dvc-files and lock-files.

Prerequisite for #4523
2020-09-09 18:44:11 +03:00
Alexander Schepanovski 67ddb7b41f Make repo lock reentrable and provide a context manager (#4416) 2020-09-08 17:40:34 +03:00
Ruslan Kuprieiev 61bf1194c3 tests: add gs etag
Kudos to @skshetry
2020-09-08 09:29:56 +03:00
Alexander Schepanovski e1f43965a2 Skip no data plots while rendering (#4541) 2020-09-07 23:25:43 +03:00
Peter Rowlands (변기호) d1cc143ef2 diff: add state for files missing from both workspace and cache (#4530)
* diff: support missing cache status for workspace diff

* diff: add --hide-missing option to hide cache missing status

* update command tests

* update diff func tests

* only show missing cache status when diffing local workspace

* suppress all output if no changes and --hide-missing

* use hide_missing instead of show_missing

* use is_dvc
2020-09-07 04:44:04 +03:00
Ruslan Kuprieiev 93a298343f run: consolidate run-cache logic (#4535)
Existing lock files are just an edge case of run-cache.

Prerequisite for #4223
2020-09-06 17:46:25 +03:00
Ruslan Kuprieiev 636a019289 dvc: compute dir hash without external cache (#4528)
Fixes #4144
2020-09-05 16:31:34 +03:00
Victor 79e8e4edaf dvc: override replace in HTTPURLInfo to include extra_parts (#4517)
* dvc: override replace in HTTPURLInfo to include extra_parts

The objective is to passtrough any parameter configured in http remotes.

fixes #4508

* add unit test for #4508

* Use HTTPURLInfo directly instead of decorator
2020-09-03 15:47:18 +03:00
Peter Rowlands (변기호) 1f495d66dd diff: use RepoTree to compare directory contents (#4518)
* diff: use RepoTree to get granular results for dir outs

* update tests for dir file granularity

* diff: prefer repo_tree.get_hash when possible

* DvcTree: handle extra files in dirty dvc out dirs
2020-09-03 15:12:57 +03:00
Saugat Pachhai 4564e8be30 support subrepos for get/list/api/import/ls (#4465)
* support subrepos

This adds support for get/list/api/import/ls on a subrepo.

Also fixes #3180, by adding a granular url support for `dvc.api.get_url`
And, of course it fixes #3369.

* Add tests

* Fix tests, started to get lock_error

* Fix tests for windows

* try to fix mac builds

* Fix comments, readd removed comments

* Fix test

* address review suggestions:

* use `spy` instead of `wraps`
* remove dvc fixture in a test
* set `cache_types` on Git repo
2020-09-03 13:47:14 +03:00
Ruslan Kuprieiev 4901765cd4 get/import: fix bug when working with mixed dirs (#4516)
Fixes #4510
2020-09-02 05:53:09 +03:00
Ruslan Kuprieiev 9887de6a65 dvc: separate cache and tree hashes (#4511)
Related to #4144 , #3069 , #1676
2020-09-02 00:30:02 +03:00
Saugat Pachhai 747cb649f7 tree.repo: fix race condition when walking nested subrepos (#4505) 2020-09-01 07:11:21 +03:00
Ruslan Kuprieiev fc321683d8 dvc: use HashInfo to interact with cache (#4506) 2020-09-01 03:32:51 +03:00
Ruslan Kuprieiev 56500601bd deps/outs: get rid of self.info (#4502)
By itself `self.info` is quite confusing, as it is not clear what it is
about. Using `hash_info` makes much more sense and is required to
support alternative hash types.

Related to #4144, #3069, #1676
2020-08-31 22:46:18 +03:00
karajan1001 777a23afac Keep status output format consistents (#4490)
fix#4482
According to the discussion. Make `DVC status [outputs]` output format the same with `DVC status [stages]` instead of `DVC status -c [outputs]`
2020-08-31 00:03:21 +03:00
Peter Rowlands (변기호)andRuslan Kuprieiev e2b5ab996f experiments: make dvc exp show more consistent w/viewer (#4489)
* experiments: cleanup show UI

* show git tag/branch names instead of SHAs by default
* make timestamp formatting more like the viewer

* update tests

* place timestamps in their own column

* update command test

* exp: remove unused regex

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-08-31 00:02:57 +03:00
Ruslan Kuprieiev fcbbfac7f8 dvc: show "Having any troubles?" only on unexpected errors (#4496)
* dvc: dont catch generic exceptions

* dvc: show "Having any troubles?" only on unexpected errors

Fixes #4126
2020-08-30 21:08:12 +03:00
Ruslan KuprieievandSaugat Pachhai 1a35128457 dvc: use HashInfo (#4495)
* dvc: use HashInfo

Related to #4144 , #3069 , #1676

* Update dvc/tree/s3.py

Co-authored-by: Saugat Pachhai <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
2020-08-30 21:07:57 +03:00
Ruslan Kuprieiev 4013256c91 external: avoid unnecessary hash computation (#4486)
* dvc: use cache state when saving downloaded cache

Related to #4485

* tree: dont compute hash from scratch for output dirs

Fixes #4485

* tree: fix get_hash
2020-08-29 21:15:44 +03:00
Peter Rowlands (변기호) d6bb4341dd plots: add dvc plots -e/--experiment option (#4488)
* plots: add -e/--experiment option for plots diff

* experiments: add method to get baseline for a given experiment

* plots: default to diff against baseline instead of workspace/head for experiments

* use exact match for experiments RE since name-rev may return relative
commits
2020-08-29 00:57:28 +03:00
Ruslan Kuprieiev a61291039e tests: use s3 client instead of resource 2020-08-28 05:16:06 +03:00
Ruslan KuprieievandJorge Orpinel b9d685a650 s3: provide more helpful messages on common errors (#4480)
* s3: provide more helpful messages on common errors

Fix #4478

* utils: introduce error_link

* Update dvc/tree/s3.py

Co-authored-by: Jorge Orpinel <[email protected]>

* fix tests

* dont forget the link

* remove redundant message

Co-authored-by: Jorge Orpinel <[email protected]>
2020-08-27 23:09:12 +03:00
Saugat Pachhai 334556f07d tests: speedup lock tests (#4471)
* tests: speedup lock tests

Patched to reduce timeout from default 5s to 0.1s

* Reduce timeout further to 0.01s
2020-08-27 05:31:25 +03:00
Saugat Pachhai 2e47674a46 tests: install hook just before using it (#4472)
Before, we were installing hooks at the start which was making checkouts
slower. With this change, it should improve these tests by twice as much.
2020-08-26 20:24:28 +03:00
Puneetha Pai c31ec3038a diff: Add sort logic to sort diff ouput irrespective of output format… (#4459)
* diff: Add sort logic to sort diff ouput irrespective of output format(default/json/md)

* diff: test stub using actual value, not pass by reference for (dict)
2020-08-26 19:35:06 +03:00
nik123 1d9acdc8f3 status: no warnings for missing state (#4470) 2020-08-26 17:17:43 +03:00
Alexander SchepanovskiandRuslan Kuprieiev bc7d952e88 Speed up dvc files load (#4417)
* Speed up dvc files load

See `._load()` implementation. Alternative is not using 3 io calls, but
simply use open and catch exceptions.

Reading is slow for old git revisions.

* tree: git: cache tree and objects

* external: ignore PermissionError on Windows

This is the same thing we have in tests.

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-08-26 05:44:27 +03:00
Ruslan Kuprieiev 718c9c98ce serialization: dont rely on the default hash type (#4466)
Prerequisite for #4144 and #3069
2020-08-25 18:34:09 +03:00
karajan1001andSaugat Pachhai fcca6361ed Outputs as target supporting for dvc status (#4433)
* Outputs as target supporting for `dvc status`

fix #4191
1. Add a related test which would fail on current version.

* Pass the tests

1. add deps to the tests.
2. make change to pass the tests.

* Update tests/func/test_status.py

Co-authored-by: Saugat Pachhai <[email protected]>

* Solve some change request.

Co-authored-by: Saugat Pachhai <[email protected]>
2020-08-25 13:51:27 +02:00
Peter Rowlands (변기호) 714503a9e8 experiments: only show queued experiments for the current commit (#4462) 2020-08-25 14:33:46 +03:00
Peter Rowlands (변기호) 01a6580b3d experiments: fix inconsistent dvc exp checkout behavior (#4461)
* experiments: stage workspace changes before using stash apply

* add test for dvc exp checkout
2020-08-25 14:33:14 +03:00
Ruslan Kuprieiev 6c352eeaf0 hdfs: implement isdir and isfile (#4460)
Fixes #4332
2020-08-25 02:02:33 +03:00
Peter Rowlands (변기호) f789cdff89 experiments: fix bug where wrong git branch checked-out before pull (#4450) 2020-08-24 15:47:25 +03:00
Paweł Redzyński a82d6e4173 metrics: don't raise on dir during collection (#4443) 2020-08-24 02:55:20 +03:00
Puneetha Pai 05c1e76091 diff command: output in markdown format with hash (#4435)
* diff: Add tests to hash formatting function

* diff: Add test for show_md with hash

* diff: Add stub test for _show_md from command
2020-08-24 02:25:09 +03:00
Saugat Pachhai 1c14a7e202 metadata() on RepoTree (#4434)
* WIP: metadata api on RepoTree

* Make outs optional

* add tests + misc. changes

* reorder tests and fields
2020-08-21 17:34:35 +03:00
Peter Rowlands (변기호) dac17e247d experiments: abstract executor tmpdir population using tree upload (#4432)
* executor: abstract tmpdir population using tree upload

* experiments: unmark flaky test

* fix wrong tree download

* LocalTree: use os.replace instead of os.rename in _upload
2020-08-21 17:32:39 +03:00
Peter Rowlands (변기호) a2a6f8bd58 tree: add overwrite parameter in tree.upload (#4442)
* tree: add overwrite flag for tree.upload

* tests: update tests

* fix test file permissions

* default to overwrite on upload
2020-08-21 17:30:27 +03:00
Saugat Pachhai de0fe94cd0 Make serializing utils use tree, misc changes (#4425)
* Also fixes plots not getting dumped on flow style
* Make serialization utils aware of trees
* Directly load yamls/tomls using utils, rather than using parser
2020-08-20 20:38:17 +03:00
Saugat Pachhai 5d1ad03fe9 git: do not ask for gpg signoff on merge (#4430) 2020-08-20 13:47:19 +03:00