1881 Commits
Author SHA1 Message Date
Saugat PachhaiandRuslan Kuprieiev fc6999ad17 Run CI for Python3.9 (#4647)
* Run CI for Python3.9, skips hdfs tests for 3.9

Also, `dvc[hdfs]` install will be a no-op till we have
a pyarrow wheel for 3.9.

Note that this does not add an entry to the classifier
claiming it supports 3.9, that will be added when 3.9
gets released (or, wait for pyarrow to be released).

* Skip HDFS related tests on Python3.9

Also, fix the python version regex for `3.9.0b5` like versions.

* Exclude python3.9 for Windows

libxml does not have wheels for Python39.
We do compile them on Linux and Mac

* Update tests/unit/test_info.py

* Use realpath for gitignore

Windows tests for Python3.8 was failing
intermittently.

Note: Also removed the realpath wrap around self.root_dir as it's already a
realpath when `Git` is instantiated.

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-10-01 10:41:39 +03:00
Saugat Pachhai e0b3e98039 api.open: fix error if uncached file inside a dir is accessed (#4652)
Wrap path in Repo.open_by_relpath() with PathInfo.
Closes #4650
2020-10-01 10:40:56 +03:00
Ruslan Kuprieiev d7a35b9852 gha: enable hdfs tests (#4644) 2020-09-30 05:43:34 +03:00
Raymond GasperandRuslan Kuprieiev b0dbcdcdb6 metrics show: allow falsey values (#4642)
* metrics show: allow falsey values

Fixes #4630

* metrics show: bug in logging to console. Fixes #4630

* repro: fix incorrect usage of _show_metrics

* fix formatting/styling

* metrics: add a simple func test

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-09-30 04:53:06 +03:00
Ruslan Kuprieiev 1fcb94dee8 hdfs: add a progress bar (#4643) 2020-09-30 03:36:50 +03:00
Ruslan Kuprieiev 7b590e7c5d gdrive: revert back to the hack for uploading empty files (#4632)
* gdrive: bring back the progress workaround

* WIP cloud: test empty files
2020-09-29 13:49:49 +03:00
Ruslan Kuprieiev e3a7857b13 repo: move index_dir to RemoteIndex (#4631) 2020-09-29 03:32:33 +03:00
Ruslan Kuprieiev de24794cd5 repo: move templates to plots (#4629) 2020-09-29 00:38:11 +03:00
Ruslan Kuprieiev 1ea8cc9e54 metrics/plots: support non-dvc repos (#4627) 2020-09-28 23:06:22 +03:00
Pavel DitenbirandRuslan Kuprieiev 738dfd49cf import: add '--file' option (#4622)
* import: add '--file' option

Fixes #4093

* Update dvc/command/imp.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-09-27 20:19:08 +03:00
Guillaume COTER a5ab01390f remote[ssh]: Add the parameter allow_agent (#4623)
This parameter is passed to paramiko.
See http://docs.paramiko.org/en/stable/api/client.html?highlight=allow_agent#paramiko.client.SSHClient.connect for more details.

Fixes #4186
2020-09-27 01:09:00 +03:00
Robert Lowry c94ab359b6 cli: add --cd flag to change directories before executing commands (#4566)
* cli: add -C flag to change directories before executing commands

Fixes #4025

* add deprecation notice to dvc repro help message
2020-09-25 04:20:22 +03:00
Peter Rowlands (변기호) 32bb46f34f experiments: support cloning parent in detached-head state (#4611)
* experiments: account for cloning detached-head parent repo

* add test for detached-head parent
2020-09-24 18:36:51 +03:00
Alexanderandaandrusenko e928034fac add py support for ParamsDependency (#4456)
* add py support for ParamsDependency

* add py support by ast

* add support modify py params

Co-authored-by: aandrusenko <[email protected]>
2020-09-24 17:25:44 +03:00
Ruslan Kuprieiev 4a82974da4 run-cache: respect cache.shared config option (#4596)
Fixes #4595
2020-09-24 01:00:53 +03:00
Ruslan Kuprieiev 688da04d4a tests: cache test directories (#4601)
* tests: ignore more hdfs tests

* tests: remove unused mark

* gha: don't limit the number of test workers

* cache created test dirs
2020-09-23 23:22:27 +03:00
Paweł Redzyński e1b34442aa plot: accept any viable target in dvc repo (#4602) 2020-09-23 17:02:12 +03:00
Ruslan Kuprieiev 38dd1c9fe7 tests: allow failure for hdfs tests (#4600) 2020-09-23 00:53:32 +03:00
Paweł Redzyński b31bd1cb30 github actions tests: initial (#4469)
* github actions tests: initial

* build: tests: add codecov action
2020-09-22 23:05:06 +03:00
Paweł Redzyński 2b3c30cb0c metrics: accept any viable target in DvcRepo (#4590) 2020-09-22 22:53:55 +03:00
Ruslan Kuprieiev cf5d461f4d api: enter state when trying to get a hash (#4584)
`get_hash` is an internal api, so conventions should be respected.

Fixes #4575
2020-09-21 23:37:51 +03:00
Robert LowryandRuslan Kuprieiev 15488ebefd scm: add config option to stage files automatically (#4543)
* scm: add "autostage" config option to stage files automatically

Fixes #4330

* Update tests/unit/scm/test_scm.py

* Update tests/unit/scm/test_scm.py

* scm_context, git: call reset_tracked_files from scm_context, remove output message

* Update dvc/repo/scm_context.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-09-21 11:59:25 +03:00
Peter Rowlands (변기호) 6a3c516b71 experiments: support committing to existing experiment branches (#4573)
* experiments: stash optional branch to extend

* experiments: support commit to existing experiment branch
2020-09-19 21:49:40 +03:00
01abf7c19c Fix error during external outputs checkout (#4570)
* Fix error during external outputs checkout

User in discord complained that external outputs' checkout
was failing. Looks like, during refactor of `HashInfo`,
this was forgotten, and as no tests existed before, we had
a regression from 1.6.5 onwards.

* fix tests

* Restyled by black (#4571)

Co-authored-by: Restyled.io <[email protected]>

Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>
Co-authored-by: Restyled.io <[email protected]>
2020-09-15 11:07:09 +03:00
karajan1001 5e69e29b2f check-ignore show sub-repo patterns (#4564)
* `check-ignore` show sub-repo patterns

fix #4555
1. Sub-repo pattern now would be considered in `check-ignore` command.
2. New outputs for sub-repo patterns in a `detail`
3. Related tests.

* message change
2020-09-15 06:59:41 +03:00
Ruslan Kuprieiev 8c19492ba6 tree: validate resolved remote:// sections (#4569)
* tree: validate resolved remote:// sections

* dvc: use checksum_jobs instead of hash_jobs

This is a regression introduced during one of the refactorings.
2020-09-14 12:15:38 +03:00
Ruslan Kuprieiev 424396b068 tree: resolve deep remote:// urls (#4568)
Fixes #4556
2020-09-14 05:49:54 +03:00
karajan1001andRuslan Kuprieiev 8b16af965c Print dvc version info for debugging (#4536)
* Print dvc version info for debugging

fix #4095
1. seperate info from dvc version.
2. print dvc version info when some unexpected error occured.

* Psutil NoneType error

* Update tests/func/test_version.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Move `package` outside dvc/utils/pkg.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-09-13 04:08:20 +03:00
Ruslan Kuprieiev cf7e534e2d tree: repo: handle dirty state (#4554) 2020-09-11 17:25:33 +03:00
Saugat Pachhai 2931d30ca5 dvc: change dvcfile.dump to do synchronized dump by default (#4544)
The API of `Dvcfile.dump` has been changed to
`dump(stage, update_pipeline=True, update_lock=True)`,
which will make it to always dump by default to both files,
and provide flags to suppress one or the other.

Fixes #4235
2020-09-10 13:12:00 +05:45
Robert F. DickersonandRuslan Kuprieiev 7fa90e425d Allow config option for HTTP method for uploads (Artifactory) (#4553)
* allow for a config option for http method

* add tests

* fix tests with method

* Update dvc/tree/http.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-09-10 05:11:41 +03:00
Victor 54a48e3256 Feature/http ssl verify configuration (#4547)
* add test for http ssl_verify configuration

* add ssl_verify flag to HTTPTree
2020-09-09 19:27:43 +03:00
Ruslan Kuprieiev 1ca2dc19c2 tree: open: handle dirty state in RepoTree (#4548)
We used to use that logic in DvcTree back before we had proper trees.
Now we have a RepoTree, that combines git/workspace tree with the
DvcTree and dirty state clearly belongs to the git/workspace and not
the dvc itself, as dvc state only depends on dvc-files and lock-files.

Prerequisite for #4523
2020-09-09 18:44:11 +03:00
Alexander Schepanovski 67ddb7b41f Make repo lock reentrable and provide a context manager (#4416) 2020-09-08 17:40:34 +03:00
Ruslan Kuprieiev 61bf1194c3 tests: add gs etag
Kudos to @skshetry
2020-09-08 09:29:56 +03:00
Alexander Schepanovski e1f43965a2 Skip no data plots while rendering (#4541) 2020-09-07 23:25:43 +03:00
Peter Rowlands (변기호) d1cc143ef2 diff: add state for files missing from both workspace and cache (#4530)
* diff: support missing cache status for workspace diff

* diff: add --hide-missing option to hide cache missing status

* update command tests

* update diff func tests

* only show missing cache status when diffing local workspace

* suppress all output if no changes and --hide-missing

* use hide_missing instead of show_missing

* use is_dvc
2020-09-07 04:44:04 +03:00
Ruslan Kuprieiev 93a298343f run: consolidate run-cache logic (#4535)
Existing lock files are just an edge case of run-cache.

Prerequisite for #4223
2020-09-06 17:46:25 +03:00
Ruslan Kuprieiev 636a019289 dvc: compute dir hash without external cache (#4528)
Fixes #4144
2020-09-05 16:31:34 +03:00
Victor 79e8e4edaf dvc: override replace in HTTPURLInfo to include extra_parts (#4517)
* dvc: override replace in HTTPURLInfo to include extra_parts

The objective is to passtrough any parameter configured in http remotes.

fixes #4508

* add unit test for #4508

* Use HTTPURLInfo directly instead of decorator
2020-09-03 15:47:18 +03:00
Peter Rowlands (변기호) 1f495d66dd diff: use RepoTree to compare directory contents (#4518)
* diff: use RepoTree to get granular results for dir outs

* update tests for dir file granularity

* diff: prefer repo_tree.get_hash when possible

* DvcTree: handle extra files in dirty dvc out dirs
2020-09-03 15:12:57 +03:00
Saugat Pachhai 4564e8be30 support subrepos for get/list/api/import/ls (#4465)
* support subrepos

This adds support for get/list/api/import/ls on a subrepo.

Also fixes #3180, by adding a granular url support for `dvc.api.get_url`
And, of course it fixes #3369.

* Add tests

* Fix tests, started to get lock_error

* Fix tests for windows

* try to fix mac builds

* Fix comments, readd removed comments

* Fix test

* address review suggestions:

* use `spy` instead of `wraps`
* remove dvc fixture in a test
* set `cache_types` on Git repo
2020-09-03 13:47:14 +03:00
Ruslan Kuprieiev 4901765cd4 get/import: fix bug when working with mixed dirs (#4516)
Fixes #4510
2020-09-02 05:53:09 +03:00
Ruslan Kuprieiev 9887de6a65 dvc: separate cache and tree hashes (#4511)
Related to #4144 , #3069 , #1676
2020-09-02 00:30:02 +03:00
Saugat Pachhai 747cb649f7 tree.repo: fix race condition when walking nested subrepos (#4505) 2020-09-01 07:11:21 +03:00
Ruslan Kuprieiev fc321683d8 dvc: use HashInfo to interact with cache (#4506) 2020-09-01 03:32:51 +03:00
Ruslan Kuprieiev 56500601bd deps/outs: get rid of self.info (#4502)
By itself `self.info` is quite confusing, as it is not clear what it is
about. Using `hash_info` makes much more sense and is required to
support alternative hash types.

Related to #4144, #3069, #1676
2020-08-31 22:46:18 +03:00
karajan1001 777a23afac Keep status output format consistents (#4490)
fix#4482
According to the discussion. Make `DVC status [outputs]` output format the same with `DVC status [stages]` instead of `DVC status -c [outputs]`
2020-08-31 00:03:21 +03:00
Peter Rowlands (변기호)andRuslan Kuprieiev e2b5ab996f experiments: make dvc exp show more consistent w/viewer (#4489)
* experiments: cleanup show UI

* show git tag/branch names instead of SHAs by default
* make timestamp formatting more like the viewer

* update tests

* place timestamps in their own column

* update command test

* exp: remove unused regex

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-08-31 00:02:57 +03:00
Ruslan Kuprieiev fcbbfac7f8 dvc: show "Having any troubles?" only on unexpected errors (#4496)
* dvc: dont catch generic exceptions

* dvc: show "Having any troubles?" only on unexpected errors

Fixes #4126
2020-08-30 21:08:12 +03:00