Commit Graph
5998 Commits
Author SHA1 Message Date
Saugat Pachhai f6d40948d6 codecov: add target coverage, and threshold for patch and project (#4873)
* Reduce codecov target to be 85%

* Increase threshold for patch and project
2020-11-10 21:47:59 +05:45
Michal NowikowskiandRuslan Kuprieiev 4b27fd7c23 [#4136] fixed recalculating relative paths in config (#4802)
* [#4136] fixed recalculating relative paths in config

* Update dvc/config.py

* Update dvc/config.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-11-10 14:56:44 +02:00
Saugat Pachhai 4220baa602 parametrization: Support keys of the dictionary to be interpolated (#4869) 2020-11-10 17:32:58 +05:45
Ruslan Kuprieiev 2e9fa0607e cache: refactor cache types (#4866) 2020-11-09 21:28:46 +02:00
Ruslan Kuprieiev 685e76cfbe gitignore: add config.ini
Generated by innosetup.
2020-11-09 13:25:45 +02:00
Saugat Pachhai dca48fe8d7 Skip loading globally used params section again locally for stages (#4859)
We were still trying to load params file used in `use`
for local stages which was throwing overwrite error.
Thanks to @dmpetrov for pointing out.
2020-11-07 09:30:12 +05:45
Saugat Pachhai 9f1330a387 Fix merge_update not converting data to Node object (#4846)
We wrap every data to one of the `Node` object. But, since we were
passing `self.data` instead of `self`, it was skipping the conversion
and instead was just trying to set data as-is, which would have make
our latter assumptions incorrect, as other codebases assume it to be
a `Node`.

I have added asserts in a few places. Also, I noticed that, on a few
test assumptions, when `resolving`, we were not converting values
back to it's original form (`CtxDict` -> `dict`, `CtxList` -> `list`
and `Value` -> _). So, I have added `node.value` property, as it is
a complement of the above fix.

This caused a ripple effect for the changes in the API as `foreach`
now resolved and changed the data to the original form,
which prevents us from tracking the data later in the "field".
So, `resolve`/`select` got a real `unwrap` option that should
reliably work for all kinds of `Node` objects.
2020-11-07 03:02:24 +02:00
Paweł Redzyński f4ee0e16b4 plots/metrics: fix some communication disrepancies (#4837)
* plots/metrics: unified parsing errors communication

* plots/metrics: unify no metrics/plots messaging

* fixup

* fixup
2020-11-07 02:56:14 +02:00
Ruslan Kuprieiev 4531aa34ca dvc: use DirInfo instead of dicts/lists (#4851)
Currently we are converting dir_info to/from lists all the time.
The reason is that dir_info is stored as list of dicts in *.dir files,
but that makes it hard to work with. In addition to that, we will likely
be changing .dir file format in the near future #829, so we need to
abstract away dir_info into something that we won't care how it will be
stored on disk.

Related #3256
Related #4847
2020-11-07 02:55:05 +02:00
Peter Rowlands (변기호) 80e181f2fc experiments: add dvc exp gc (#4803)
* experiments: add `dvc exp gc`

* tests: add exp gc tests

* add exp gc command test

* fix review issue
2020-11-06 19:05:41 +09:00
Peter Rowlands (변기호) c85a46cdda experiments: add checkpoint output type (#4845)
* output: add checkpoint output type

* stage: handle checkpoint outs

* run/repro: handle checkpoint outs

* experiments: use checkpoint output type in tests

* tests: update run command tests
2020-11-06 14:44:21 +09:00
Ruslan Kuprieiev 89bf7fce13 dvc: limit aws-sam-translator version for moto (#4853)
moto has aws-sam-translator as an indirect dependency that since 1.29.0
started to have a public package
that causes conflicts in flufl.lock when it tries to use atpublic
package.

```
tests/func/test_add.py:21: in <module>
    from dvc.main import main
dvc/main.py:7: in <module>
    from dvc import analytics
dvc/analytics.py:16: in <module>
    from dvc.lock import Lock, LockError
dvc/lock.py:8: in <module>
    import flufl.lock
/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/flufl/lock/__init__.py:3: in <module>
    from flufl.lock._lockfile import (
/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/flufl/lock/_lockfile.py:42: in <module>
    from public import public
E   ImportError: cannot import name 'public'
```

Oddly enough, this seems to also been causing pylint issues. public is a
terrible name for a package SMH
2020-11-06 04:10:54 +02:00
Ruslan Kuprieiev 42d50f084f gs: don't forget to include file size (#4852) 2020-11-06 02:59:05 +02:00
Paweł Redzyński edd8344659 plots: fix plots path when calling from different dir (#4843) 2020-11-05 08:11:24 +01:00
Ruslan Kuprieiev 4b17685351 cache: refactor dir_info conversion (#4844) 2020-11-04 21:42:14 +02:00
Ruslan Kuprieiev e1b82c5222 dvc: add size/nfiles for deps/outs (#4836)
* dvc: add size for deps/outs

Related to #3256

* dvc: add nfiles for deps/outs

* dvc: put size/nfiles into the hash_info
2020-11-04 19:22:57 +02:00
Saugat Pachhai e9a9415fbf refactor: move interpolation logic inside the Context (#4840)
* Move interpolation logic to the Context

* Don't unwrap by default on `Context.select`
2020-11-04 17:04:40 +05:45
Sami Jawhar 6494bd0151 Plots: Better confusion matrix, and normalized version to (#4775)
* Add normalized confusion plot, and add text labels

* Impute missing XY combinations
2020-11-04 11:33:11 +01:00
Liew XunandSaugat Pachhai d1441ee16a http: raise error on unsuccessful http statuses on exists (#4785)
* http: added check for HTTP status code 404

Fixes #4288

* http: add unit test and minor change

In tests_exists, changed to raise a HTTPError from dvc instead.

Fixes #4288

* http: removed fixtures in test for HTTPTree.exists

Fixes #4288

* test: add comments on why `raw` is required on the mock

Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2020-11-04 14:43:42 +05:45
Ruslan Kuprieiev 7da3de451f build: test built binary (#4839) 2020-11-03 23:30:41 +02:00
Ruslan Kuprieiev 4a3f06df3a oss: require pycryptodome<3.9.9
3.9.9 doesn't have windows wheels https://github.com/Legrandin/pycryptodome/issues/465
2020-11-03 20:28:44 +02:00
Ruslan Kuprieiev 2225764438 tests: set subprocess._active to None (#4818)
To better reflect https://github.com/python/cpython/pull/14360
2020-11-03 19:31:04 +02:00
Peter Rowlands (변기호) 644444b5a7 experiments: fix checkpoint checkout issues (#4823)
* checkout: add option to suppress missing file warnings

* experiments: fix checkpoints branch checkout issues

* experiments: _get_branch_containing should not change scm checkout
2020-11-03 22:21:54 +09:00
Saugat Pachhai 60b69b93cb Add flaky-service (#4834)
Use master branch for flaky-service

Generate TAP results during pytest
2020-11-03 18:41:10 +05:45
Saugat Pachhai fec1499c67 Move interpolation logic to the Context (#4833) 2020-11-03 17:42:28 +05:45
Saugat Pachhai a38941b47e Refactor context set/alias to Context class (#4831) 2020-11-03 17:06:46 +05:45
Saugat Pachhai 6a9ab9cdfb dvc.yaml: implement set keyword (#4757) 2020-11-02 20:32:07 +05:45
Saugat Pachhai e211423398 Implement foreach ... in loop in dvc.yaml (#4734)
* Implement foreach...in loop in dvc.yaml

* fix tests
2020-11-02 18:56:23 +05:45
Peter Rowlands (변기호) 96cdd073bd diff: handle empty output status (#4826) 2020-11-02 21:16:29 +09:00
simonlousky 59d5c73c5e increased http timeout to 60 seconds (#4820) 2020-11-01 13:19:55 +02:00
Kevin Lloyd Bernal 58bed0e9ba import: add --no-exec param (#4817)
To be consistent with the same --no-exec param in import-url.

Fixes #4567.
2020-10-31 20:03:33 +02:00
Ruslan Kuprieiev 92752e0392 dvc: upgrade dulwich to 0.20.11
This one has wheels for python 3.9
2020-10-30 19:16:43 +02:00
João Vitor 4a7bb49bb0 command: add dvc push instruction after dvc repro (#4737)
* command: add dvc push instruction after dvc repro

Adds a logger.info() statement to remind the user to run dvc push after
running dvc repro.

Fixes #2359

* command: change dvc push instruction for dvc repro

* command: Use backticks and else in dvc repro log
2020-10-30 15:47:00 +01:00
Peter Rowlands (변기호) 3d58f4b8ce git: move branch iteration functionality from experiments into scm.git (#4810)
* git: move branch iteration functionality from experiments into scm.git

* add branch_revs unit test
2020-10-30 19:06:35 +09:00
Peter Rowlands (변기호)andPaweł Redzyński f827d641d5 add: warn on cache link errors instead of failing (#4801)
* add: warn on cache link failure instead of error

* add test case

* fix linted string

Co-authored-by: Paweł Redzyński <[email protected]>

Co-authored-by: Paweł Redzyński <[email protected]>
2020-10-30 02:01:11 +02:00
Paweł Redzyński 9940540917 metrics/plots: test metrics collection on no target on revision (#4789)
* metrics: include revision information when collecting metrics files

* plots: test warning message on no plot on revision

* plots/metrics: move to one package, introduce common tests

* tests: fix utils placement
2020-10-30 02:00:09 +02:00
Ruslan Kuprieiev 32fa8e33c4 git: use dulwich for .gitignore (#4809)
In my tests `Git._ignore` is taking around ~65seconds for the whole test
suit(~50K invocations). The reason is that `gitpython` is launching a real
`git check-ignore` process, that takes time and resources.

Switching to `pygit2` or `dulwich` makes it drop to ~5seconds, saving us
almost a minute of the test run time and a lot of resources by not
launching any processes. I've chosen `dulwich` over `pygit2` here
because it is much more pythonic and as long as the performance is
on-par with `pygit2` it is better for us to use `dulwich`.
2020-10-30 01:01:58 +02:00
Peter Rowlands (변기호) 911534116e experiments: fix stash ordering bug (#4805) 2020-10-29 18:01:34 +09:00
Ruslan Kuprieiev 9b136f0c3f pyarrow: upgrade to 2.0.0 (#4795) 2020-10-27 19:14:30 +02:00
Ruslan Kuprieiev 411b388960 dvc: bump to 1.9.1 1.9.1 2020-10-27 17:33:27 +02:00
Juliano Efson Sales e6016ca202 tests: testing when overwriting files (#4738) 2020-10-27 15:52:16 +02:00
Peter Rowlands (변기호) b2495aa230 add: fix issue when adding already tracked symlinked files (#4778)
* LocalOutput: paths symlinked outside of git can't be gitignored

* utils: contains_symlink_up_to should only check parent dirs on add

* tests: update test_add symlink cases

* add: forbid adding symlinked dirs or files inside symlinked dirs

* update tests

* add troubleshooting link
2020-10-27 15:39:01 +02:00
f4522760ba --jobs now takes effect on imported data (#4765)
* `--jobs` now takes effect on imported data

fix #4692

* Tests for this change

* Solve issue on python3.6 and 3.7

* Update tests/unit/test_external_repo.py

* Update tests/unit/test_external_repo.py

* Update tests/unit/test_external_repo.py

* Update tests/unit/test_external_repo.py

* tests: fix formatting

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-10-27 01:21:51 +02:00
Ruslan Kuprieiev 0736c64188 gha: don't create gs creds file if no creds 2020-10-27 00:23:34 +02:00
Paweł Redzyński 6cf194f85f plot: fix no metrics data detection (#4764) 2020-10-26 20:05:55 +02:00
Paweł Redzyński 5038fd225b plots/metrics: support show for no repo (#4761) 2020-10-26 20:04:49 +02:00
Ruslan Kuprieiev bfecd7d8e8 gha: add gs creds (#4791)
* gha: add gs creds

Related to #4418

* tests: gs: don't use gcloud
2020-10-26 19:20:58 +02:00
Ruslan Kuprieiev a5bdfa90f0 gha: add aws creds (#4790)
Related to #4418
2020-10-26 17:02:07 +02:00
Paweł Redzyński 4d75e75631 test_update_import: roll back skip (#4781) 2020-10-26 15:28:30 +02:00
Tomás Farías Santana f8ba5daae2 Replace dict calls for literals (#4784)
* Replace dict calls for literals

* Pin flake8-comprehensions version to 3.3.0
2020-10-25 19:51:22 +02:00