Commit Graph
19 Commits
Author SHA1 Message Date
karajan1001 76b31ac282 fix #3588 (#3770)
* fix #3588

remove improper edges from output DAG.

* less code modification

* deepsource required

* deepsource required

* refactoring for reducing cognitive complexity.
2020-05-09 21:00:46 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
Ruslan Kuprieiev bc7f177c7a run: use multistage dvcfiles by default (#3740)
Adding hidden `--single-stage` to simplify the test migration.
2020-05-04 19:55:39 +03:00
Ruslan Kuprieiev f9088e1c67 dvc: rename pipelines.yaml -> dvc.yaml (#3710) 2020-04-30 18:19:51 +03:00
Saugat PachhaiandRuslan Kuprieiev 1937527249 dvc: implement multistage dvcfile (#3676)
* dvc: use lockfile as yaml instead of json

* refactor Dvcfile into Pipeline file and Single stage file

* fix tests

* dvc: fix outputs

* add more tests for collection of outputs

* add tests for data cloud/get/import/ls

* tests: test for checkouts

* Allow other checksums other than md5

* tests: use iterdir instead of os.listdir

* cleanup errors reported by cc and ds

* utils: throw DvcException instead of plain Exception

* tests: use yaml.load instead of json.load

* run: split assignments

* Update dvc/stage/exceptions.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-04-28 13:36:05 +03:00
Saugat Pachhai 128fa7f887 dvc: add a support for multistage dvcfile. (#3584)
Now, pipeline stage files can house multiple stages, and separate lockfiles
are created which has the checksums, whereas Dvcfile will be clean and human
readable and editable. The *.dvc files will be generated for output files.

It is available via a hidden flag: -n.

Fixes #1871
PR: #3584
2020-04-17 22:58:36 +03:00
Ruslan Kuprieiev 7edee656da pipeline: fix --outs output (#3507)
Fixes #3497
2020-03-18 23:03:41 +02:00
Ruslan Kuprieiev 06c634adee pipelines: fix edges gathering (#3421) 2020-03-05 14:16:57 +02:00
Fábio Santos 758205a52a PR feedback 2020-02-03 18:45:44 +00:00
Fábio Santos 11ad0a9803 remove __ 2020-02-03 18:41:55 +00:00
Fábio Santos a7bad88159 remove unnecessary chdir 2020-01-28 11:49:24 +00:00
Fábio Santos 233790ced5 add test and traverse nodes when showing outs 2020-01-23 17:54:00 +00:00
Paweł Redzyński 945eb181bc pipeline: test: migrate to dir helpers 2020-01-08 11:23:43 +01:00
Mr. Outis fd82402d34 py3: use new magic from PEP 3135 2020-01-02 16:24:31 -06:00
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Ruslan Kuprieiev 381601f4ee dvc: cache graph/stages/pipelines (#2529)
* dvc: cache graph/stages/pipelines

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: check_dag: don't reset the cache

https://github.com/iterative/dvc/pull/2529#discussion_r327711928

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: rename check_dag(stages) -> check_modified_graph(new_stages)

https://github.com/iterative/dvc/pull/2529#discussion_r327708439

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: refactor "active" logic and remove it from Repo class

* repo: use tree.setter to automate cache reset

As noted by @Suor's , this is much less prone to errors.

* repo: move graph-related staticmethods into graph.py

As per @Suor's request.

* dvc: remove unused from_directory

https://github.com/iterative/dvc/pull/2529#discussion_r328079258

* repo: use _tree instead of __tree

https://github.com/iterative/dvc/pull/2529#discussion_r328071115

* repo: add a note about cache invalidation on repo.lock release

https://github.com/iterative/dvc/pull/2529#discussion_r328071635

* repo: rename reset() -> _reset()

https://github.com/iterative/dvc/pull/2529#discussion_r328079899

* pipeline: use graph helpers

https://github.com/iterative/dvc/pull/2529#discussion_r327502293

* repo: use list comprehension for collect()

https://github.com/iterative/dvc/pull/2529#discussion_r328076651

* repo: graph: only use first cycle

https://github.com/iterative/dvc/pull/2529#discussion_r328081346

* dvc: don't use G.nodes()

https://github.com/iterative/dvc/pull/2529#discussion_r328089671
https://github.com/iterative/dvc/pull/2529#discussion_r328089878
2019-09-26 07:42:29 +03:00
Ruslan Kuprieiev 918ef646dc appveyor: test py2 and py3
Related to #2166

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-23 13:33:47 +03:00
TomHortons e3de8aac65 pipeline: show: dot: print dot 2019-05-22 11:11:23 +09:00
Ruslan Kuprieiev 6f11f1f51f test: move func tests to tests/func
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-16 20:19:22 +03:00