Commit Graph
53 Commits
Author SHA1 Message Date
Saugat Pachhai 5101bf1253 gc: make it safer by only activating by use of certain flags (#3363)
* gc: make it safer by only activating by use of certain flags

Previous plain `gc` is now not supported. The same behavior can be trigger by
`gc -w` or `gc --workspace`. When `--all-{tags, commits, branches}` are
specified, it will work as if `-w` is also specified (same as before).

* tests: adjust gc to use workspace on test_repro
2020-02-21 12:39:59 +02:00
Ruslan Kuprieiev 6474725d4f test: repro: prevent dvc.stages resetting (#3368) 2020-02-20 03:45:28 +02:00
Alexander Schepanovski 6ccae711fe dvc: refactor config (#3298)
* dvc: refactor config

Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:

    with config.edit("local") as conf:
        conf["core"]["remote"] = "new-default"

Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict

* test: do not set cache.type for remotes that doesn't support it

* dvc: fix windows absolute paths validation in config

* test: fix windows separate drive test

* dvc: stylistic fixes in config
2020-02-13 19:35:37 +02:00
Saugat Pachhai 7993600019 test: use helper remotes in external repro 2020-01-14 23:31:44 +05:45
Saugat Pachhai 26d85f1a52 test: merge gcp related functions to GCP helper class 2020-01-13 23:49:22 +05:45
Saugat Pachhai d55427dbd9 test: refactor hdfs related functions in HDFS test helpers 2020-01-13 16:30:15 +05:45
Saugat Pachhai 141f308a6a test: refactor ssh related functions in SSH and SSHMocked test helpers 2020-01-13 14:23:42 +05:45
Ruslan Kuprieiev 2a9d19f116 tests: use mocked ssh 2020-01-11 03:14:45 +02:00
Paweł Redzyński 0554cbfe9c repro: tests: use Path-like objects in repro_dir fixture 2020-01-09 11:41:44 +01:00
Paweł Redzyński 93c54763fe repro: tests: yield dict of stages from repro_dir 2020-01-09 11:10:41 +01:00
Paweł Redzyński 25437ca600 repro: tests: refactor for readibility 2020-01-08 21:46:39 +01:00
Paweł Redzyński e93504668d repro: tests: migrate to dir helpers 2020-01-08 21:35:29 +01:00
Ruslan Kuprieiev 1dcce1d8b0 rwlock: check that .dvc/lock is locked
Mostly a sanity check, but we did have a potential bug in `destroy`,
whcih this PR fixes along the way.

Kudos @shcheklein for catching this one.
2020-01-06 22:34:34 +02:00
Saugat Pachhai 78dd4896ca test: merge _should_test_aws and _get_aws_url inside S3 test helper 2020-01-06 22:08:28 +05:45
Mr. Outis fd82402d34 py3: use new magic from PEP 3135 2020-01-02 16:24:31 -06:00
Mr. Outis 27f581cf4e py3: remove imports from utils.compat 2019-12-31 23:16:11 -06:00
Mr. Outis 6d36b85736 py3: remove string compat related code 2019-12-30 23:49:19 -06:00
Mr. Outis 94a0914bba py3: get rid of future imports 2019-12-30 19:35:48 -06:00
Alexander Schepanovski b6b07dde8b dvc: make graph consist of stages not their paths 2019-12-30 12:07:52 +07:00
Saugat Pachhai 42991d3b75 Move more stuff from data_cloud to tests/remotes.py 2019-11-30 10:04:45 +05:45
Paweł Redzyński 7218ff202c remote: http: rename http error, refactor test 2019-11-29 15:30:55 +01:00
Paweł Redzyński 72abfb0572 remote: http: raise exception when download response with error status code 2019-11-29 15:30:55 +01:00
Saugat Pachhai 461a79e1e8 Fix typos 2019-11-26 20:07:20 +05:45
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Ruslan Kuprieiev 46d221fade test: s3: try using boto3.client 2019-11-03 22:38:48 +02:00
Ruslan Kuprieiev 59e62a7227 checkout: accept multiple targets (#2640)
Make `Repo.checkout` accept a list of targets, instead of having to feed
it targets one-by-one. This allows us to properly handle checkout
errors.

Fixes #2580
2019-10-21 17:06:21 +03:00
Ruslan Kuprieiev 803ffaec57 remote: local: don't auto-create directory in __init__
Imagine the scenario of shared cache. To set it up, user would do:

```
dvc config cache.dir /path
dvc config cache.shared group
```

Now, when he runs any dvc operation that involves the Repo instance,
RemoteLOCAL will also be initialized and it will currently auto-create
the directory with a regular `os.mkdir`, which is not using group
permissions. To solve that, we can't simply use `self.makdirs` instead,
as it will cause same issues if you run any dvc command before
`dvc config cache.shared group`. So what we should do is create
directories on demand, instead of creating it in advance, same as we do
in other remotes.

The change itself is pretty trivial and involved removing that `mkdir`
line, adjusting tests to not rely on cache dir existence and also
adjusting `dvc version` to not rely on cache directory existance.

Fixes #2563
2019-10-14 15:50:49 +03:00
Ruslan Kuprieiev 381601f4ee dvc: cache graph/stages/pipelines (#2529)
* dvc: cache graph/stages/pipelines

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: check_dag: don't reset the cache

https://github.com/iterative/dvc/pull/2529#discussion_r327711928

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: rename check_dag(stages) -> check_modified_graph(new_stages)

https://github.com/iterative/dvc/pull/2529#discussion_r327708439

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: refactor "active" logic and remove it from Repo class

* repo: use tree.setter to automate cache reset

As noted by @Suor's , this is much less prone to errors.

* repo: move graph-related staticmethods into graph.py

As per @Suor's request.

* dvc: remove unused from_directory

https://github.com/iterative/dvc/pull/2529#discussion_r328079258

* repo: use _tree instead of __tree

https://github.com/iterative/dvc/pull/2529#discussion_r328071115

* repo: add a note about cache invalidation on repo.lock release

https://github.com/iterative/dvc/pull/2529#discussion_r328071635

* repo: rename reset() -> _reset()

https://github.com/iterative/dvc/pull/2529#discussion_r328079899

* pipeline: use graph helpers

https://github.com/iterative/dvc/pull/2529#discussion_r327502293

* repo: use list comprehension for collect()

https://github.com/iterative/dvc/pull/2529#discussion_r328076651

* repo: graph: only use first cycle

https://github.com/iterative/dvc/pull/2529#discussion_r328081346

* dvc: don't use G.nodes()

https://github.com/iterative/dvc/pull/2529#discussion_r328089671
https://github.com/iterative/dvc/pull/2529#discussion_r328089878
2019-09-26 07:42:29 +03:00
Ruslan Kuprieiev 384fd4cd9a test: repro: fix link types 2019-09-10 02:12:28 +03:00
Mr. Outis 73133e8c7a tests: [re] fix hardlink stuff with HDFS 2019-09-06 12:35:38 -05:00
Mr. Outis f97c33178f tests: use copy as cache type for HDFS 2019-09-06 12:05:53 -05:00
Alexander Schepanovski 50b5b87304 dvc: batch push/pull/fetch/status
This will speed up cache collection and then run single cloud op for all
collected cache.

Fixes #2308.
2019-07-30 13:33:01 +03:00
Ruslan Kuprieiev 641edacef6 import-url: lock by-default
Fixes #2238

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-22 20:24:00 +03:00
Ruslan Kuprieiev e215364e6d test: http: use random port (#2269)
Fixes #1878

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-15 13:18:10 +03:00
Alexander Schepanovski 732fd01cc5 test: drop a reimport (#2247) 2019-07-10 03:24:59 +03:00
Alexander Schepanovski 90b1a80240 test: test ssh dir output (#2246) 2019-07-10 03:24:35 +03:00
Ruslan Kuprieiev fcc8938b33 test: repro: fix formatting 2019-07-08 21:27:32 +03:00
Ruslan Kuprieiev e8b0d2d987 test: disable repro s3 test on windows 2019-07-08 21:24:23 +03:00
Ruslan Kuprieiev f3c4aff6c5 test: don't use python to call awscli 2019-07-08 17:44:18 +03:00
Ruslan Kuprieiev af34867d96 test: try calling awscli through python 2019-07-06 02:49:08 +03:00
Jorge Orpinel fba7d85cbd dvc: Use "DVC-file" term as much as possible
Keep "stage file" in the context of pipelines

No issue
2019-06-26 14:05:18 -05:00
Ruslan Kuprieiev dd6ed7e7e9 test: s3: remove debug marker 2019-06-25 13:13:26 +03:00
Ruslan Kuprieiev dceca73f00 tests: debug appveyor s3 failure 2019-06-25 05:26:45 +03:00
Ruslan Kuprieiev 918ef646dc appveyor: test py2 and py3
Related to #2166

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-23 13:33:47 +03:00
Ruslan Kuprieiev 1ba8e0171d dvc: replace pkg concept with external repo
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-21 01:16:00 +03:00
prd 0615ff8730 pathinfo: extract relative path method to standalone method 2019-06-18 12:08:19 +02:00
Alexander Schepanovski d80293c44d dvc: path info refactoring
Replaced Path* dataclasses with PathInfo/URLInfo/CloudURLInfo, which
know how parse, serialize, stringify and join themselves. They also
provide useful accessors like part of an url or pathlib.Path like
.parent or .name.

Along the way two families of bugs where fixed and should never occur in
the future:
- paths not matched as strings anymore, e.g. using str.startswith()
- Output.path is eliminated as ambiguous

Both families might had caused undefined surprising behavior when local
and remote paths clash.
2019-06-11 19:19:17 +07:00
David Příhoda d072c2bd79 Fix after renaming dvc fixture to dvc_repo. 2019-05-27 14:14:44 +02:00
David Příhoda e11c3cbbc5 Add repro -R, support dirs & stage files when running pull/push/repro -R. 2019-05-27 12:48:46 +02:00
pawel 01c2f6c174 tests: rename dvc repo fixture 2019-05-21 15:04:43 -07:00