Commit Graph
42 Commits
Author SHA1 Message Date
skshetry e7d4f5b24e tests: filter warnings (#9115) 2023-03-06 15:41:45 +05:45
skshetry 2a2c02c184 remove typing.cast (#9082) 2023-02-26 13:15:54 +05:45
skshetry a86470ccbc update black and change line-length to 88 (#8953) 2023-02-04 05:28:07 +02:00
skshetry 7ab4c59d70 test: run pytest-style checks (#8901) 2023-01-28 14:10:53 +05:45
David de la Iglesia CastroandPeter Rowlands dd2d2dce19 param: Fix status check for Tuples.
Closes #8803

Update dvc/dependency/param.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2023-01-19 08:19:16 +01:00
Ruslan Kuprieiev a92ace0054 deps: bump dvc-data to 0.31.3 2023-01-16 01:16:35 +02:00
skshetry 309ae4d995 mypy: enable check-untyped-defs (#8815) 2023-01-15 21:40:40 +05:45
Peter Rowlands c29e721823 output: support loading/dumping dir trees from metadata 2022-09-15 13:13:31 +03:00
Ruslan Kuprieiev a124d7e866 deps: bump dvc-data to 0.6.3 2022-09-07 02:23:29 +03:00
Alex HallandDavid de la Iglesia Castro 2ca1d6c7c1 refactor: extract load_path function from common code using LOADERS (#7983)
Following up on https://github.com/iterative/dvc/pull/7965#discussion_r912684074

Co-authored-by: David de la Iglesia Castro <[email protected]>
2022-08-09 09:09:43 +00:00
Saugat Pachhai (सौगात) af649af462 deps: bump dvc-data to 0.0.6; fix imports 2022-06-14 17:50:57 +05:45
Daniele Trifirò 793f97bdd5 fix dvc-objects/data imports 2022-06-13 18:03:53 +05:45
Ruslan Kuprieiev 9ac2f49b57 Revert "fix dvc-objects/data imports"
This reverts commit 677da2b22c.
2022-06-10 12:01:07 +03:00
Daniele Trifirò 677da2b22c fix dvc-objects/data imports 2022-06-10 14:07:33 +05:45
Saugat Pachhai (सौगात) 37921d618d dvc: remove dvc.objects, fix imports 2022-05-20 14:02:08 +05:45
Saugat Pachhai (सौगात) 14c9348f93 dvc.objects: move hash_info, state and meta into dvc.objects 2022-05-11 17:35:48 +03:00
Saugat Pachhai (सौगात) 9d96ddfbe7 exp init: create params file by default
Note that this won't add support for tracking the params file,
it'll only create the params file. Some changes on error and
prompts are made.
2022-03-21 15:00:20 +05:45
Saugat Pachhai (सौगात) a3bab48f41 params: allow tracking file 2022-03-21 13:47:44 +05:45
Saugat Pachhai (सौगात) 943481f0a3 exp init: handle exception properly on missing parameters file 2022-01-24 17:43:29 +05:45
Batuhan Taskaya 9aec8f1926 dvc: get rid of path_info
Intermediate PR in removing path_info from our code base.

It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.

For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).

Fixes #6774
2021-11-10 00:52:48 +02:00
Ruslan Kuprieiev 32467ada32 objects: extract size/nfiles to Meta (#6784)
* stage: capture metadata

* tree: capture metadata

* objects: extract size/nfiles from HashInfo/HashFile/Tree to Meta
2021-10-12 03:06:18 +03:00
Paweł Redzyński 21af7438db plots/metrics/params/experiments show: stop throwing exceptions (#5984)
* params/metrics/plots/exp show: introduce onerror

* fixup

* plots/metrics/params: removing errors

* removing more errors

* removing errors

* some more removing

* new plots ux

* new ux for metrics

* new ux for params

* fixing tests

* fixups

* minor test fixes

* minor test fixes

* commands: plots/params: fix ui imports

* working on error handlng decorator, stopped at show checkpoints

* error_handler, further dvelopment

* plots just before moving loading to collect

* appending errors to results: initial passing

* fixup

* onerror: remove onerror fixture

* fixups

* plots/metrics/params/experiments: temporarily remove error warnings

* plots: extract parsing out of class method

* metrics/experiments: provide exception serializer for show-json

* params/plots/metrics: signal errors

* rebase fixups

* fix TODOs

* fixup

* skshetry's review refactor
2021-07-16 15:36:04 +02:00
Ruslan Kuprieiev 8a7ccab82f fs: deps/outs: get rid of types (#6054)
Pre-requisite for removing fs.path_info and switching to regular strings
as paths in filesystems.

With new filesystems we no longer need the zoo of different dependency
and output types, as most of them are now just handled by the underlying
filesystem. Though deps/outs are still due for a refactor in other
aspects.
2021-05-25 19:25:09 +03:00
Saugat Pachhai b9000ed926 pre-commit: update black to 21.5b1 (#6032)
* pre-commit: update black to 21.5b1

also updated others, but black is more noticeable

* remove magic trailing comma
2021-05-18 15:51:15 +00:00
Paweł RedzyńskiandJorge Orpinel c39f91ecf4 params: read from ParamsDependency if possible (#5550)
* params: read from ParamsDependency if possible

* params: introduce include_untracked flag

* params: include untracked upon targets

* experiments: add option to show tracked params

* fixing test

* params: rename tracked to deps

* params: roll back to default params.yaml behavior

* experiments: add option to show only used params on diff

* Update dvc/command/experiments.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/experiments.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/params.py

Co-authored-by: Jorge Orpinel <[email protected]>

Co-authored-by: Jorge Orpinel <[email protected]>
2021-04-12 17:08:02 +02:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
Ruslan Kuprieiev 56500601bd deps/outs: get rid of self.info (#4502)
By itself `self.info` is quite confusing, as it is not clear what it is
about. Using `hash_info` makes much more sense and is required to
support alternative hash types.

Related to #4144, #3069, #1676
2020-08-31 22:46:18 +03:00
Saugat Pachhai de0fe94cd0 Make serializing utils use tree, misc changes (#4425)
* Also fixes plots not getting dumped on flow style
* Make serialization utils aware of trees
* Directly load yamls/tomls using utils, rather than using parser
2020-08-20 20:38:17 +03:00
Saugat Pachhai 14745039a4 Move to yaml1.2 (#4415)
* use yaml 1.2 everywhere

This commit:
* gets rid of pyyaml
* uses ruamel everywhere
* tries to unite toml and yaml

However, this will require a few more changes that will come soonish.
Eg: Current `parse_yaml` and `parse_toml` behavior could be error-prone,
so, it's better to introduce `tree` inside these serialization methods
(which I will try to tackle next).

Another is, yaml load can be slow for json. We could speed it up just by
using `json.load(s)` here (also in upcoming patch).

* Fix tests
2020-08-19 17:56:38 +03:00
dtrifiroandDaniele Trifirò e2b1fb1f64 add toml support for ParamsDependency (#4258)
* add toml support for ParamsDependency

* add toml support to dvc show

Co-authored-by: Daniele Trifirò <[email protected]>
2020-07-24 19:09:04 +03:00
Saugat Pachhai fe1d7e9f1d params: fix skipping of params dvc.lock when it's a falsy value (#4185)
The falsy values were being skipped when merging params values
from dvc.yaml and dvc.lock, showing the values as always changed.

These values were never added to `info`, but were found in
`self.params` making them appear as changed.

Fixes #4184
2020-07-09 21:29:26 +03:00
Saugat Pachhai 99b58d3d05 refactor/test multistage load for params and outputs (#3949)
* refactor multistage load for params and outputs

* tests: load params

* tests: output loading from pipeline file

* fix test

* fix typo in name

* split params load

* rename params func s/inject_values/fill_values

* fix tests

* simplify loads_params and output.load_from_pipeline

* address @pared's suggestions for tests
2020-06-04 23:35:07 +03:00
Saugat Pachhai b8aa8ad951 pyupgrade: 3.6+ (#3805)
* pyupgrade: 3.6+

* tests: pyupgrade 3.6+
2020-05-15 16:48:02 +03:00
Peter Rowlands (변기호) 833e82a7ea Drop Python 3.5 support (#3792)
* drop python 3.5 support

* dvc: remove fspath() use inside os.path calls

* dvc: replace dvc.compat.fspath with os.fspath

* tests: replace dvc.compat.fspath with os.fspath

* utils: relpath() should use fspath/str

* utils: os.readlink doesn't support path-like obj on Windows until 3.8
2020-05-14 01:46:42 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
Ruslan KuprieievandSaugat Pachhai 827c994e74 stage: cache: use lockfiles (#3713)
* stage: cache: use lockfiles

* reorganize

* Reorganize, use fill_from_lock to load stage cache

* load params from build cache

* adjust tests

Co-authored-by: Saugat Pachhai <[email protected]>
2020-05-01 15:23:11 +03:00
nik123 e03128a8de refactor: dvc/dependency class names unification (#3688) 2020-04-28 12:49:09 +03:00
Ruslan Kuprieiev c1ac6bb06b dvc: introduce dvc params diff (#3573)
* params: use repo.tree

* remote: local: use `repo.tree` for `exists()`

* param: handle missing params

* stage: move status warning to debug

* dvc: introduce `dvc params diff`

Fixes #3528
2020-04-09 05:18:49 +03:00
Ruslan Kuprieiev 9de8d1e0fa params: use dpath for xpaths (#3531)
Currently `dvc run -p angles` is broken when you use it with `params.yaml`
that looks like this:

```
angles:
    - 2
    - 3
    - 100
    - 3.2
    - 36.2
    - 3.6
```

because we are incorrectly using `flatten` instead of simply using some
xpath-like library to access params that are specified.
2020-03-25 06:17:09 +02:00
Helge Munk JacobsenandRuslan Kuprieiev 28748ff345 introduce hyper parameters and config (#3515)
* Work in progress

* added file parsing and name validation + adjust schema

* Exceptions on bad input

* Support multiple parameters

* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

* fix formatting

* dep: param: fix schema

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* run: check if --params is specified

* dvc: rework param handling

* params: fix exc formatting

* params: support basic nested configs

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-03-24 17:56:39 +02:00
Ruslan Kuprieiev 8f546f253b Revert "Restyle #3393 introduce hyper parameters and config (#3516)" (#3517)
This reverts commit 65d22c7dbd.
2020-03-20 00:54:21 +02:00
65d22c7dbd Restyle #3393 introduce hyper parameters and config (#3516)
* Work in progress

* added file parsing and name validation + adjust schema

* Exceptions on bad input

* Support multiple parameters

* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

* Restyled by black

Co-authored-by: elgehelge <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
2020-03-20 00:53:41 +02:00