Commit Graph
11 Commits
Author SHA1 Message Date
Saugat Pachhai af2a12a188 Use 'schema' top-level element in the lockfile for version (#5222) 2021-01-06 20:08:14 +02:00
Saugat Pachhai 0cc5227f47 Add versioning scheme in the dvc.lock (#5128)
* Add versioning scheme in the dvc.lock

* remove whitespace

* use actual size in tests assumptions
2021-01-06 15:44:19 +05:45
Ruslan Kuprieiev e1b82c5222 dvc: add size/nfiles for deps/outs (#4836)
* dvc: add size for deps/outs

Related to #3256

* dvc: add nfiles for deps/outs

* dvc: put size/nfiles into the hash_info
2020-11-04 19:22:57 +02:00
Saugat Pachhai 14745039a4 Move to yaml1.2 (#4415)
* use yaml 1.2 everywhere

This commit:
* gets rid of pyyaml
* uses ruamel everywhere
* tries to unite toml and yaml

However, this will require a few more changes that will come soonish.
Eg: Current `parse_yaml` and `parse_toml` behavior could be error-prone,
so, it's better to introduce `tree` inside these serialization methods
(which I will try to tackle next).

Another is, yaml load can be slow for json. We could speed it up just by
using `json.load(s)` here (also in upcoming patch).

* Fix tests
2020-08-19 17:56:38 +03:00
Saugat Pachhai e31364979b Move toml/yaml serialization to dvc.utils.serialize (#4411) 2020-08-18 13:01:16 +05:45
Saugat Pachhai d201fad1e3 tests: serialize pipeline file (#3985)
serialize: make params ordered
2020-06-09 14:22:55 +03:00
Saugat Pachhai 219662597b refactor: rename utils.stage to utils.yaml (#3923) 2020-06-01 16:49:46 +03:00
Saugat Pachhai 9897f7c470 test: metric/cache/persist keys (#3785) 2020-05-15 13:58:50 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
Ruslan KuprieievandSaugat Pachhai 827c994e74 stage: cache: use lockfiles (#3713)
* stage: cache: use lockfiles

* reorganize

* Reorganize, use fill_from_lock to load stage cache

* load params from build cache

* adjust tests

Co-authored-by: Saugat Pachhai <[email protected]>
2020-05-01 15:23:11 +03:00
Saugat PachhaiandRuslan Kuprieiev 22c60dd608 dump: deterministic lockfile dump (#3711)
* dump: lockfile is dumped deterministically

The dump is no longer deterministic/dependent on the pipeline file,
but is sorted based on file names in outs, deps or params.
Also, the params inside each files are also sorted based on name.
However, the objects inside params are not sorted deterministically
as I think it's too much to sort that, and is not easy (considering
the types of objects it might hold, eg: lists, objects, etc).

This will also provide ordered dumps for Python3.5

* fix windows 3.8 test

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-04-30 22:34:06 +03:00