Commit Graph
63 Commits
Author SHA1 Message Date
nik123 1e65098819 refactor: dvc/output class names unification (#3695) 2020-04-28 22:29:32 +03:00
Peter Rowlands (변기호) 4fdf7ca7fd repo: use reverse post-order DFS in repro --downstream (#3689)
* tests: ensure repro --downstream preserves dependency order

* repo: use reverse post-order DFS in repro --downstream

- Fixes #3602

* make test_repro and test_repro_multistage consistent

* rebase master, update multistage test
2020-04-28 18:17:10 +03:00
Saugat PachhaiandRuslan Kuprieiev 1937527249 dvc: implement multistage dvcfile (#3676)
* dvc: use lockfile as yaml instead of json

* refactor Dvcfile into Pipeline file and Single stage file

* fix tests

* dvc: fix outputs

* add more tests for collection of outputs

* add tests for data cloud/get/import/ls

* tests: test for checkouts

* Allow other checksums other than md5

* tests: use iterdir instead of os.listdir

* cleanup errors reported by cc and ds

* utils: throw DvcException instead of plain Exception

* tests: use yaml.load instead of json.load

* run: split assignments

* Update dvc/stage/exceptions.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-04-28 13:36:05 +03:00
nik123 fefbcc0b4c refactor: dvc/remotes name unification (#3684)
Partially fixes #2089
2020-04-26 20:55:00 +03:00
Peter Rowlands (변기호) 7e9ff62d89 flake8: fix ambiguous var names (#3674) 2020-04-24 13:35:34 +09:00
Saugat Pachhai 63c466d821 refactor: extract stage loading outside of dvcfile (#3655)
* refactor: extract stage loading outside of dvcfile

* repo: simplify stage collection

* fix typing
2020-04-20 23:55:02 +03:00
Saugat Pachhai 128fa7f887 dvc: add a support for multistage dvcfile. (#3584)
Now, pipeline stage files can house multiple stages, and separate lockfiles
are created which has the checksums, whereas Dvcfile will be clean and human
readable and editable. The *.dvc files will be generated for output files.

It is available via a hidden flag: -n.

Fixes #1871
PR: #3584
2020-04-17 22:58:36 +03:00
Saugat PachhaiandRuslan Kuprieiev 65854c4946 refactor: extract stage's relationship with Dvcfile out from Stage (#3619)
* refactor stage into dvcfile

* Split create_stage out of dvcfile to Stage::create() and overwrite

* Remove logging from Repo::run

* tests: stop using --remove-outs

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-04-12 23:09:29 +03:00
Ruslan Kuprieiev 63fe2893a8 run: remove deprecated --cwd (#3623)
It has been deprecated for a long time now and it significantly
complicates the code, hiding multiple bugs.
2020-04-11 19:57:05 +03:00
Ruslan Kuprieiev 6559b9d222 dvc: use protected mode by default (#3472)
* dvc: use protected mode by default

Fixes #3261
Fixes #2041

* unprotect: adjust help message

https://github.com/iterative/dvc.org/pull/1058/files/21aab371a487acf6f6e6201b29cd832e7c55ed23#r393324668
2020-03-17 19:16:21 +02:00
Saugat Pachhai 5101bf1253 gc: make it safer by only activating by use of certain flags (#3363)
* gc: make it safer by only activating by use of certain flags

Previous plain `gc` is now not supported. The same behavior can be trigger by
`gc -w` or `gc --workspace`. When `--all-{tags, commits, branches}` are
specified, it will work as if `-w` is also specified (same as before).

* tests: adjust gc to use workspace on test_repro
2020-02-21 12:39:59 +02:00
Ruslan Kuprieiev 6474725d4f test: repro: prevent dvc.stages resetting (#3368) 2020-02-20 03:45:28 +02:00
Alexander Schepanovski 6ccae711fe dvc: refactor config (#3298)
* dvc: refactor config

Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:

    with config.edit("local") as conf:
        conf["core"]["remote"] = "new-default"

Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict

* test: do not set cache.type for remotes that doesn't support it

* dvc: fix windows absolute paths validation in config

* test: fix windows separate drive test

* dvc: stylistic fixes in config
2020-02-13 19:35:37 +02:00
Saugat Pachhai 7993600019 test: use helper remotes in external repro 2020-01-14 23:31:44 +05:45
Saugat Pachhai 26d85f1a52 test: merge gcp related functions to GCP helper class 2020-01-13 23:49:22 +05:45
Saugat Pachhai d55427dbd9 test: refactor hdfs related functions in HDFS test helpers 2020-01-13 16:30:15 +05:45
Saugat Pachhai 141f308a6a test: refactor ssh related functions in SSH and SSHMocked test helpers 2020-01-13 14:23:42 +05:45
Ruslan Kuprieiev 2a9d19f116 tests: use mocked ssh 2020-01-11 03:14:45 +02:00
Paweł Redzyński 0554cbfe9c repro: tests: use Path-like objects in repro_dir fixture 2020-01-09 11:41:44 +01:00
Paweł Redzyński 93c54763fe repro: tests: yield dict of stages from repro_dir 2020-01-09 11:10:41 +01:00
Paweł Redzyński 25437ca600 repro: tests: refactor for readibility 2020-01-08 21:46:39 +01:00
Paweł Redzyński e93504668d repro: tests: migrate to dir helpers 2020-01-08 21:35:29 +01:00
Ruslan Kuprieiev 1dcce1d8b0 rwlock: check that .dvc/lock is locked
Mostly a sanity check, but we did have a potential bug in `destroy`,
whcih this PR fixes along the way.

Kudos @shcheklein for catching this one.
2020-01-06 22:34:34 +02:00
Saugat Pachhai 78dd4896ca test: merge _should_test_aws and _get_aws_url inside S3 test helper 2020-01-06 22:08:28 +05:45
Mr. Outis fd82402d34 py3: use new magic from PEP 3135 2020-01-02 16:24:31 -06:00
Mr. Outis 27f581cf4e py3: remove imports from utils.compat 2019-12-31 23:16:11 -06:00
Mr. Outis 6d36b85736 py3: remove string compat related code 2019-12-30 23:49:19 -06:00
Mr. Outis 94a0914bba py3: get rid of future imports 2019-12-30 19:35:48 -06:00
Alexander Schepanovski b6b07dde8b dvc: make graph consist of stages not their paths 2019-12-30 12:07:52 +07:00
Saugat Pachhai 42991d3b75 Move more stuff from data_cloud to tests/remotes.py 2019-11-30 10:04:45 +05:45
Paweł Redzyński 7218ff202c remote: http: rename http error, refactor test 2019-11-29 15:30:55 +01:00
Paweł Redzyński 72abfb0572 remote: http: raise exception when download response with error status code 2019-11-29 15:30:55 +01:00
Saugat Pachhai 461a79e1e8 Fix typos 2019-11-26 20:07:20 +05:45
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Ruslan Kuprieiev 46d221fade test: s3: try using boto3.client 2019-11-03 22:38:48 +02:00
Ruslan Kuprieiev 59e62a7227 checkout: accept multiple targets (#2640)
Make `Repo.checkout` accept a list of targets, instead of having to feed
it targets one-by-one. This allows us to properly handle checkout
errors.

Fixes #2580
2019-10-21 17:06:21 +03:00
Ruslan Kuprieiev 803ffaec57 remote: local: don't auto-create directory in __init__
Imagine the scenario of shared cache. To set it up, user would do:

```
dvc config cache.dir /path
dvc config cache.shared group
```

Now, when he runs any dvc operation that involves the Repo instance,
RemoteLOCAL will also be initialized and it will currently auto-create
the directory with a regular `os.mkdir`, which is not using group
permissions. To solve that, we can't simply use `self.makdirs` instead,
as it will cause same issues if you run any dvc command before
`dvc config cache.shared group`. So what we should do is create
directories on demand, instead of creating it in advance, same as we do
in other remotes.

The change itself is pretty trivial and involved removing that `mkdir`
line, adjusting tests to not rely on cache dir existence and also
adjusting `dvc version` to not rely on cache directory existance.

Fixes #2563
2019-10-14 15:50:49 +03:00
Ruslan Kuprieiev 381601f4ee dvc: cache graph/stages/pipelines (#2529)
* dvc: cache graph/stages/pipelines

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: check_dag: don't reset the cache

https://github.com/iterative/dvc/pull/2529#discussion_r327711928

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: rename check_dag(stages) -> check_modified_graph(new_stages)

https://github.com/iterative/dvc/pull/2529#discussion_r327708439

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: refactor "active" logic and remove it from Repo class

* repo: use tree.setter to automate cache reset

As noted by @Suor's , this is much less prone to errors.

* repo: move graph-related staticmethods into graph.py

As per @Suor's request.

* dvc: remove unused from_directory

https://github.com/iterative/dvc/pull/2529#discussion_r328079258

* repo: use _tree instead of __tree

https://github.com/iterative/dvc/pull/2529#discussion_r328071115

* repo: add a note about cache invalidation on repo.lock release

https://github.com/iterative/dvc/pull/2529#discussion_r328071635

* repo: rename reset() -> _reset()

https://github.com/iterative/dvc/pull/2529#discussion_r328079899

* pipeline: use graph helpers

https://github.com/iterative/dvc/pull/2529#discussion_r327502293

* repo: use list comprehension for collect()

https://github.com/iterative/dvc/pull/2529#discussion_r328076651

* repo: graph: only use first cycle

https://github.com/iterative/dvc/pull/2529#discussion_r328081346

* dvc: don't use G.nodes()

https://github.com/iterative/dvc/pull/2529#discussion_r328089671
https://github.com/iterative/dvc/pull/2529#discussion_r328089878
2019-09-26 07:42:29 +03:00
Ruslan Kuprieiev 384fd4cd9a test: repro: fix link types 2019-09-10 02:12:28 +03:00
Mr. Outis 73133e8c7a tests: [re] fix hardlink stuff with HDFS 2019-09-06 12:35:38 -05:00
Mr. Outis f97c33178f tests: use copy as cache type for HDFS 2019-09-06 12:05:53 -05:00
Alexander Schepanovski 50b5b87304 dvc: batch push/pull/fetch/status
This will speed up cache collection and then run single cloud op for all
collected cache.

Fixes #2308.
2019-07-30 13:33:01 +03:00
Ruslan Kuprieiev 641edacef6 import-url: lock by-default
Fixes #2238

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-22 20:24:00 +03:00
Ruslan Kuprieiev e215364e6d test: http: use random port (#2269)
Fixes #1878

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-15 13:18:10 +03:00
Alexander Schepanovski 732fd01cc5 test: drop a reimport (#2247) 2019-07-10 03:24:59 +03:00
Alexander Schepanovski 90b1a80240 test: test ssh dir output (#2246) 2019-07-10 03:24:35 +03:00
Ruslan Kuprieiev fcc8938b33 test: repro: fix formatting 2019-07-08 21:27:32 +03:00
Ruslan Kuprieiev e8b0d2d987 test: disable repro s3 test on windows 2019-07-08 21:24:23 +03:00
Ruslan Kuprieiev f3c4aff6c5 test: don't use python to call awscli 2019-07-08 17:44:18 +03:00
Ruslan Kuprieiev af34867d96 test: try calling awscli through python 2019-07-06 02:49:08 +03:00