Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-09 23:10:24 +01:00
Code Issues Packages Projects Releases Wiki Activity
5,127 Commits 7 Branches 579 Tags
de1f67afbd93929d51c5dd052d5f8a951e77c182
Commit Graph
50 Commits
Author SHA1 Message Date
Saugat Pachhai 7993600019 test: use helper remotes in external repro 2020-01-14 23:31:44 +05:45
Saugat Pachhai 26d85f1a52 test: merge gcp related functions to GCP helper class 2020-01-13 23:49:22 +05:45
Saugat Pachhai d55427dbd9 test: refactor hdfs related functions in HDFS test helpers 2020-01-13 16:30:15 +05:45
Saugat Pachhai 141f308a6a test: refactor ssh related functions in SSH and SSHMocked test helpers 2020-01-13 14:23:42 +05:45
Ruslan Kuprieiev 2a9d19f116 tests: use mocked ssh 2020-01-11 03:14:45 +02:00
Paweł Redzyński 0554cbfe9c repro: tests: use Path-like objects in repro_dir fixture 2020-01-09 11:41:44 +01:00
Paweł Redzyński 93c54763fe repro: tests: yield dict of stages from repro_dir 2020-01-09 11:10:41 +01:00
Paweł Redzyński 25437ca600 repro: tests: refactor for readibility 2020-01-08 21:46:39 +01:00
Paweł Redzyński e93504668d repro: tests: migrate to dir helpers 2020-01-08 21:35:29 +01:00
Ruslan Kuprieiev 1dcce1d8b0 rwlock: check that .dvc/lock is locked 
Mostly a sanity check, but we did have a potential bug in `destroy`,
whcih this PR fixes along the way.

Kudos @shcheklein for catching this one.
 2020-01-06 22:34:34 +02:00
Saugat Pachhai 78dd4896ca test: merge _should_test_aws and _get_aws_url inside S3 test helper 2020-01-06 22:08:28 +05:45
Mr. Outis fd82402d34 py3: use new magic from PEP 3135 2020-01-02 16:24:31 -06:00
Mr. Outis 27f581cf4e py3: remove imports from utils.compat 2019-12-31 23:16:11 -06:00
Mr. Outis 6d36b85736 py3: remove string compat related code 2019-12-30 23:49:19 -06:00
Mr. Outis 94a0914bba py3: get rid of future imports 2019-12-30 19:35:48 -06:00
Alexander Schepanovski b6b07dde8b dvc: make graph consist of stages not their paths 2019-12-30 12:07:52 +07:00
Saugat Pachhai 42991d3b75 Move more stuff from data_cloud to tests/remotes.py 2019-11-30 10:04:45 +05:45
Paweł Redzyński 7218ff202c remote: http: rename http error, refactor test 2019-11-29 15:30:55 +01:00
Paweł Redzyński 72abfb0572 remote: http: raise exception when download response with error status code 2019-11-29 15:30:55 +01:00
Saugat Pachhai 461a79e1e8 Fix typos 2019-11-26 20:07:20 +05:45
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Ruslan Kuprieiev 46d221fade test: s3: try using boto3.client 2019-11-03 22:38:48 +02:00
Ruslan Kuprieiev 59e62a7227 checkout: accept multiple targets (#2640) 
Make `Repo.checkout` accept a list of targets, instead of having to feed
it targets one-by-one. This allows us to properly handle checkout
errors.

Fixes #2580
 2019-10-21 17:06:21 +03:00
Ruslan Kuprieiev 803ffaec57 remote: local: don't auto-create directory in __init__ 
Imagine the scenario of shared cache. To set it up, user would do:

```
dvc config cache.dir /path
dvc config cache.shared group
```

Now, when he runs any dvc operation that involves the Repo instance,
RemoteLOCAL will also be initialized and it will currently auto-create
the directory with a regular `os.mkdir`, which is not using group
permissions. To solve that, we can't simply use `self.makdirs` instead,
as it will cause same issues if you run any dvc command before
`dvc config cache.shared group`. So what we should do is create
directories on demand, instead of creating it in advance, same as we do
in other remotes.

The change itself is pretty trivial and involved removing that `mkdir`
line, adjusting tests to not rely on cache dir existence and also
adjusting `dvc version` to not rely on cache directory existance.

Fixes #2563
 2019-10-14 15:50:49 +03:00
Ruslan Kuprieiev 381601f4ee dvc: cache graph/stages/pipelines (#2529) 
* dvc: cache graph/stages/pipelines

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: check_dag: don't reset the cache

https://github.com/iterative/dvc/pull/2529#discussion_r327711928

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: rename check_dag(stages) -> check_modified_graph(new_stages)

https://github.com/iterative/dvc/pull/2529#discussion_r327708439

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: refactor "active" logic and remove it from Repo class

* repo: use tree.setter to automate cache reset

As noted by @Suor's , this is much less prone to errors.

* repo: move graph-related staticmethods into graph.py

As per @Suor's request.

* dvc: remove unused from_directory

https://github.com/iterative/dvc/pull/2529#discussion_r328079258

* repo: use _tree instead of __tree

https://github.com/iterative/dvc/pull/2529#discussion_r328071115

* repo: add a note about cache invalidation on repo.lock release

https://github.com/iterative/dvc/pull/2529#discussion_r328071635

* repo: rename reset() -> _reset()

https://github.com/iterative/dvc/pull/2529#discussion_r328079899

* pipeline: use graph helpers

https://github.com/iterative/dvc/pull/2529#discussion_r327502293

* repo: use list comprehension for collect()

https://github.com/iterative/dvc/pull/2529#discussion_r328076651

* repo: graph: only use first cycle

https://github.com/iterative/dvc/pull/2529#discussion_r328081346

* dvc: don't use G.nodes()

https://github.com/iterative/dvc/pull/2529#discussion_r328089671
https://github.com/iterative/dvc/pull/2529#discussion_r328089878
 2019-09-26 07:42:29 +03:00
Ruslan Kuprieiev 384fd4cd9a test: repro: fix link types 2019-09-10 02:12:28 +03:00
Mr. Outis 73133e8c7a tests: [re] fix hardlink stuff with HDFS 2019-09-06 12:35:38 -05:00
Mr. Outis f97c33178f tests: use copy as cache type for HDFS 2019-09-06 12:05:53 -05:00
Alexander Schepanovski 50b5b87304 dvc: batch push/pull/fetch/status 
This will speed up cache collection and then run single cloud op for all
collected cache.

Fixes #2308.
 2019-07-30 13:33:01 +03:00
Ruslan Kuprieiev 641edacef6 import-url: lock by-default 
Fixes #2238

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-22 20:24:00 +03:00
Ruslan Kuprieiev e215364e6d test: http: use random port (#2269) 
Fixes #1878

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-15 13:18:10 +03:00
Alexander Schepanovski 732fd01cc5 test: drop a reimport (#2247) 2019-07-10 03:24:59 +03:00
Alexander Schepanovski 90b1a80240 test: test ssh dir output (#2246) 2019-07-10 03:24:35 +03:00
Ruslan Kuprieiev fcc8938b33 test: repro: fix formatting 2019-07-08 21:27:32 +03:00
Ruslan Kuprieiev e8b0d2d987 test: disable repro s3 test on windows 2019-07-08 21:24:23 +03:00
Ruslan Kuprieiev f3c4aff6c5 test: don't use python to call awscli 2019-07-08 17:44:18 +03:00
Ruslan Kuprieiev af34867d96 test: try calling awscli through python 2019-07-06 02:49:08 +03:00
Jorge Orpinel fba7d85cbd dvc: Use "DVC-file" term as much as possible 
Keep "stage file" in the context of pipelines

No issue
 2019-06-26 14:05:18 -05:00
Ruslan Kuprieiev dd6ed7e7e9 test: s3: remove debug marker 2019-06-25 13:13:26 +03:00
Ruslan Kuprieiev dceca73f00 tests: debug appveyor s3 failure 2019-06-25 05:26:45 +03:00
Ruslan Kuprieiev 918ef646dc appveyor: test py2 and py3 
Related to #2166

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-23 13:33:47 +03:00
Ruslan Kuprieiev 1ba8e0171d dvc: replace pkg concept with external repo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-21 01:16:00 +03:00
prd 0615ff8730 pathinfo: extract relative path method to standalone method 2019-06-18 12:08:19 +02:00
Alexander Schepanovski d80293c44d dvc: path info refactoring 
Replaced Path* dataclasses with PathInfo/URLInfo/CloudURLInfo, which
know how parse, serialize, stringify and join themselves. They also
provide useful accessors like part of an url or pathlib.Path like
.parent or .name.

Along the way two families of bugs where fixed and should never occur in
the future:
- paths not matched as strings anymore, e.g. using str.startswith()
- Output.path is eliminated as ambiguous

Both families might had caused undefined surprising behavior when local
and remote paths clash.
 2019-06-11 19:19:17 +07:00
David Příhoda d072c2bd79 Fix after renaming dvc fixture to dvc_repo. 2019-05-27 14:14:44 +02:00
David Příhoda e11c3cbbc5 Add repro -R, support dirs & stage files when running pull/push/repro -R. 2019-05-27 12:48:46 +02:00
pawel 01c2f6c174 tests: rename dvc repo fixture 2019-05-21 15:04:43 -07:00
Mr. Outis c0104e2e61 repro: implement downstream option (#1926) 
* repro: implement downstream option

Close #1555

* Update scripts/completion/dvc.zsh

Co-Authored-By: mroutis <[email protected]>

* Update dvc/command/repro.py

Co-Authored-By: mroutis <[email protected]>
 2019-04-25 04:50:44 +03:00
Alexander Schepanovski 0c867399bb dvc: retain formatting and comments in .dvc files (#1885) 
* test: remove never used copy of load_stage_file()

* dvc: save comments and custom formatting in stage files

* Add a couple of TODOs

* test: test repro doesn't clear comments and formatting in .dvc

As part of this I started writing pytest-like tests and added commonly
used fixtures.

* test: simplify and prettify dvc formatting retained test

Also test that line comments after md5 fields are retained.

* test: fix tests

* dvc: fix stage and metrics opens not closed

* dvc: always use utf-8 in stage files

It used to encode in system default encoding. Not sure if PyYAML was
handling this silently or we were just lucky all the way.

* test: unit test apply_diff()

* test: remove TODOs and ASK

* dvc: polish dvc-formatting PR

* dvc: move stage load/dump utils to a new stage utils file

* dvc: clean up apply_diff()

* test: test stage md5 ignores comments

* test: remove pytests starting marker
 2019-04-20 01:12:53 +03:00
Ruslan Kuprieiev 6f11f1f51f test: move func tests to tests/func 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-16 20:19:22 +03:00