Commit Graph
16 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 060fc4fbb7 state: use granularly (#4676)
* state: add simple reentrance

* state: pass Repo instead of LocalCache

* state: use granularly

Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
2020-10-08 08:36:14 +03:00
Alexanderandaandrusenko e928034fac add py support for ParamsDependency (#4456)
* add py support for ParamsDependency

* add py support by ast

* add support modify py params

Co-authored-by: aandrusenko <[email protected]>
2020-09-24 17:25:44 +03:00
Ruslan Kuprieiev 688da04d4a tests: cache test directories (#4601)
* tests: ignore more hdfs tests

* tests: remove unused mark

* gha: don't limit the number of test workers

* cache created test dirs
2020-09-23 23:22:27 +03:00
Ruslan Kuprieiev 56500601bd deps/outs: get rid of self.info (#4502)
By itself `self.info` is quite confusing, as it is not clear what it is
about. Using `hash_info` makes much more sense and is required to
support alternative hash types.

Related to #4144, #3069, #1676
2020-08-31 22:46:18 +03:00
Saugat Pachhai 14745039a4 Move to yaml1.2 (#4415)
* use yaml 1.2 everywhere

This commit:
* gets rid of pyyaml
* uses ruamel everywhere
* tries to unite toml and yaml

However, this will require a few more changes that will come soonish.
Eg: Current `parse_yaml` and `parse_toml` behavior could be error-prone,
so, it's better to introduce `tree` inside these serialization methods
(which I will try to tackle next).

Another is, yaml load can be slow for json. We could speed it up just by
using `json.load(s)` here (also in upcoming patch).

* Fix tests
2020-08-19 17:56:38 +03:00
Saugat Pachhai e31364979b Move toml/yaml serialization to dvc.utils.serialize (#4411) 2020-08-18 13:01:16 +05:45
dtrifiroandDaniele Trifirò e2b1fb1f64 add toml support for ParamsDependency (#4258)
* add toml support for ParamsDependency

* add toml support to dvc show

Co-authored-by: Daniele Trifirò <[email protected]>
2020-07-24 19:09:04 +03:00
Saugat Pachhai fe1d7e9f1d params: fix skipping of params dvc.lock when it's a falsy value (#4185)
The falsy values were being skipped when merging params values
from dvc.yaml and dvc.lock, showing the values as always changed.

These values were never added to `info`, but were found in
`self.params` making them appear as changed.

Fixes #4184
2020-07-09 21:29:26 +03:00
Saugat Pachhai 99b58d3d05 refactor/test multistage load for params and outputs (#3949)
* refactor multistage load for params and outputs

* tests: load params

* tests: output loading from pipeline file

* fix test

* fix typo in name

* split params load

* rename params func s/inject_values/fill_values

* fix tests

* simplify loads_params and output.load_from_pipeline

* address @pared's suggestions for tests
2020-06-04 23:35:07 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
nik123 e03128a8de refactor: dvc/dependency class names unification (#3688) 2020-04-28 12:49:09 +03:00
Ruslan Kuprieiev c1ac6bb06b dvc: introduce dvc params diff (#3573)
* params: use repo.tree

* remote: local: use `repo.tree` for `exists()`

* param: handle missing params

* stage: move status warning to debug

* dvc: introduce `dvc params diff`

Fixes #3528
2020-04-09 05:18:49 +03:00
Ruslan Kuprieiev 9de8d1e0fa params: use dpath for xpaths (#3531)
Currently `dvc run -p angles` is broken when you use it with `params.yaml`
that looks like this:

```
angles:
    - 2
    - 3
    - 100
    - 3.2
    - 36.2
    - 3.6
```

because we are incorrectly using `flatten` instead of simply using some
xpath-like library to access params that are specified.
2020-03-25 06:17:09 +02:00
Helge Munk JacobsenandRuslan Kuprieiev 28748ff345 introduce hyper parameters and config (#3515)
* Work in progress

* added file parsing and name validation + adjust schema

* Exceptions on bad input

* Support multiple parameters

* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

* fix formatting

* dep: param: fix schema

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* run: check if --params is specified

* dvc: rework param handling

* params: fix exc formatting

* params: support basic nested configs

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-03-24 17:56:39 +02:00
Ruslan Kuprieiev 8f546f253b Revert "Restyle #3393 introduce hyper parameters and config (#3516)" (#3517)
This reverts commit 65d22c7dbd.
2020-03-20 00:54:21 +02:00
65d22c7dbd Restyle #3393 introduce hyper parameters and config (#3516)
* Work in progress

* added file parsing and name validation + adjust schema

* Exceptions on bad input

* Support multiple parameters

* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

* Restyled by black

Co-authored-by: elgehelge <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
2020-03-20 00:53:41 +02:00