Commit Graph
103 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 2305549460 erepo: fix DependencyREPO schema
URL should be required and rev or rev_lock should be optional.

Related to #3087
2020-01-21 12:58:33 +02:00
Alexander Schepanovski 033ef88a4e external: refactor external_repo() and its users
Some things are fixed along the way:
- no cache shared between incompatible funcs
- some incosistent exceptions are now consistent
- import updates for git files now works both for dvc and git repos
- "auto-generated-upstream already exists" fixed
- dvc.api.open()/read() works with git repos now

Fixes #3124, #2976. Closes #3172.
2020-01-19 21:38:06 +07:00
Sujith H 1b7eda2745 Fix import issue without remote config in the target
Fix the import issue when target does not have
remote config.

Signed-off-by: Sujith H <[email protected]>
2020-01-19 11:42:05 +05:30
Fábio Santos fcb7681683 take exception transformation into account 2020-01-16 14:49:51 +00:00
Fábio Santos cd3ef1850d simplify control flow 2020-01-16 14:44:09 +00:00
Fábio Santos a9bdaec63c rework code path; remove extraneous context generator 2020-01-16 13:37:13 +00:00
Fábio Santos a0bdaa4c9e use checksum facilities 2020-01-16 12:51:20 +00:00
Fábio Santos 4ada384fcd remove trace statement 2020-01-15 12:43:14 +00:00
Fábio Santos 008a531535 support non-DVC repositories 2020-01-15 12:39:15 +00:00
Fábio Santos 289797e7eb can now get whether an item changed, but not when an item changes from non-DVC to DVC 2020-01-15 11:33:26 +00:00
Fábio Santos 1a26d3b60f Merge remote-tracking branch 'upstream/master' into feature/2959-status-of-imported-files 2020-01-14 23:08:43 +00:00
Fábio Santos 3228b476fa compare checksums instead of file bytes 2020-01-14 22:58:11 +00:00
Ruslan Kuprieiev 7d6ecd1ccb import: don't forget to set rev_lock for git files
Related to #3087
2020-01-15 00:38:55 +02:00
Fábio Santos 8f464c2a07 status: don't clone more than once for plain files 2020-01-14 19:00:46 +00:00
Fábio Santos c9b0593220 status: enable status of files imported from remote git repos 2020-01-14 18:51:56 +00:00
Ruslan Kuprieiev f93574895f Merge pull request #3020 from chatcannon/dvc-import-regular-git
import: allow importing from non-DVC git repositories
2020-01-05 07:18:03 +02:00
Mr. Outis fd82402d34 py3: use new magic from PEP 3135 2020-01-02 16:24:31 -06:00
Mr. Outis 27f581cf4e py3: remove imports from utils.compat 2019-12-31 23:16:11 -06:00
Chris Kerr 3ceb1db5a8 import: Allow importing from non-DVC git repos
Fixes #2977

Refactor `_copy_if_git_file` to use `cached_clone` directly, so
that copying still works even if the source repo does not have a
'.dvc' subdirectory.
2019-12-31 13:19:33 +00:00
Mr. Outis 6d36b85736 py3: remove string compat related code 2019-12-30 23:49:19 -06:00
Mr. Outis 94a0914bba py3: get rid of future imports 2019-12-30 19:35:48 -06:00
Wojciech Baranowski c02c0169cf Revert "DependencyREPO.status: handle git-tracked files"
This reverts commit 48bf4490d5.
2019-12-15 18:14:47 +02:00
Wojciech Baranowski 7461f14845 Revert "_git_status(): use different cache dirs"
This reverts commit 8985759d66.
2019-12-15 18:14:38 +02:00
Wojciech Baranowski 8985759d66 _git_status(): use different cache dirs 2019-12-14 14:42:51 +02:00
Wojciech Baranowski 48bf4490d5 DependencyREPO.status: handle git-tracked files 2019-12-14 14:08:38 +02:00
Wojciech Baranowski a36ac0c4ea dependency/repo: fix py2 support 2019-12-14 14:08:38 +02:00
Wojciech Baranowski c04f5a0f9d import: allow downloading regular files/dirs tracked by git
Fixes #2862
2019-12-14 14:08:38 +02:00
Alexander Schepanovski 78fad73f2b perf: switch schema validation library
voluptuous is compiling schema to python function, which makes it
faster.
2019-11-14 22:33:28 +01:00
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Alexander Schepanovski 86811aac20 imp: dedup and parallelize fetching imported deps (#2654)
* imp: dedup and parallelize external fetches

This prevents same fetches for external artifacts, groups them by source
and fetches in parallel.

* imp: test that fetch works on non workspace deps
2019-10-23 18:02:50 +03:00
Mr. OutisandRuslan Kuprieiev ae90904fb3 pull: download data from imported stages (#2499)
* wip - pull is working

* use fetch instead of reproduce

* test pull and fetch

* style changes

* fetch: catch config error on fetch instead of get_remote

* output: remove unused guard clause

* fetch: define a more specific error instead of ConfigError

* fetch: display number of errors correctly

* exceptions: remove "Failed" prefix to download/upload

* fetch: catch other exceptions that might halt the download

* fetch: log catched exceptions

* fetch: reformat exception

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* fix formatting
2019-09-22 01:28:12 +03:00
Paweł RedzyńskiandRuslan Kuprieiev 53f6aead95 Update dvc/dependency/base.py
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
2019-09-14 19:40:59 +02:00
pawel 0935930f46 output: prevent stage file from being used as output 2019-09-13 13:31:50 +02:00
Ruslan Kuprieiev ecb28a285c Merge pull request #2307 from efiop/2238
import-url: lock by-default
2019-07-23 01:44:04 +03:00
Ruslan Kuprieiev 641edacef6 import-url: lock by-default
Fixes #2238

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-22 20:24:00 +03:00
Ruslan Kuprieiev 4a0b3a807d import: only donwload specific output
Fixes #2252

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-22 15:04:10 +03:00
Ruslan Kuprieiev 932b0d7af9 introduce dvc update for ext repos (#2218)
* erepo: dvc: implement `update` for imports from ext repos

Fixes #1774
Fixes #2139
Fixes #2201

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* deps: repo: make  private

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* update: only support repo imports

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-06 02:28:35 +03:00
Ruslan Kuprieiev 1ba8e0171d dvc: replace pkg concept with external repo
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-21 01:16:00 +03:00
Ruslan Kuprieiev 5941ace270 dvc: initial 'dvc pkg' implementation
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-17 18:12:36 +03:00
Alexander Schepanovski d80293c44d dvc: path info refactoring
Replaced Path* dataclasses with PathInfo/URLInfo/CloudURLInfo, which
know how parse, serialize, stringify and join themselves. They also
provide useful accessors like part of an url or pathlib.Path like
.parent or .name.

Along the way two families of bugs where fixed and should never occur in
the future:
- paths not matched as strings anymore, e.g. using str.startswith()
- Output.path is eliminated as ambiguous

Both families might had caused undefined surprising behavior when local
and remote paths clash.
2019-06-11 19:19:17 +07:00
Ruslan Kuprieiev 9e9e078de3 config: move logic from CLI to API
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-06 14:41:02 +03:00
Ruslan Kuprieiev a593a9ce2e dvc: refactor PathInfos
Following same conventions from remotes/outsputs/dependencies.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-12 17:18:46 +03:00
Ruslan Kuprieiev d1881fd1b1 remote: separate http and https
Fixes #1988

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-12 16:39:40 +03:00
pared 3ff10005fa path_info: remove dict path_info, introduce PathInfo class 2019-05-10 22:25:50 -07:00
Ruslan Kuprieiev 00cfb04fea remote: generalize file/dir handling
Fixes #1654
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Mr. Outis d355b1e66e config: allow cross-referenced remotes
Close #1614
2019-04-08 22:10:03 -05:00
Ruslan Kuprieiev ba8d83e9b1 dvc: refactor dvc project class
1) split Project's methods into separate files;
2) rename Project into Repo(similar to gitpython's Repo)

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-13 03:31:23 +02:00
Ruslan Kuprieiev 4896f08ddf dvc: comply with both flake8 and black
flake8 provides some additional features like unused vars check, etc, so
it makes total sense to use it as well.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-04 00:10:36 +02:00
Ruslan Kuprieiev 4069e34130 dvc: use black instead of flake8
Fixes #1546

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-30 03:48:36 +02:00
Mr. Outis 0d46921ec7 py2: add unicode literals
Fix #1512
2019-01-27 20:40:21 -06:00