Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
3,511 Commits 7 Branches 579 Tags
3098460f567707fad5d5d63015f60e9798e83d61
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev 06d347178f remote: base: remove whitespace 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-13 02:25:15 +03:00
Ruslan Kuprieiev c308ad8a58 remote: base: don't forget about the progress bar 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-13 02:08:29 +03:00
Ruslan Kuprieiev baa548eec9 remote: base: call makedirs in move 
Remotes like s3/gs/etc don't need to create dirs in advance.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-10 04:53:50 +03:00
Ruslan Kuprieiev 919a49a231 travis: windows: cache chocolatey downloads 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-10 04:16:11 +03:00
Ruslan Kuprieiev eb3b3f9f4e travis: windows: skip install stage 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-10 04:15:22 +03:00
Ruslan Kuprieiev 3f19036646 travis: osx: speedup install stage 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-10 04:15:22 +03:00
Ruslan Kuprieiev 2b36d25f49 remote: base: remove unused argument 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev 9955f1c64f utils: merge isinstance checks 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev 640e737570 remote: base: remove unused argument 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev 24428e47c3 remote: local: cache_exists: match parameters from the BASE 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev 713e288adc remote: base: checkout: remove unnecessary return 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev b2fe3f7a71 utils: remove unnecessary else 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev 9c5fd0c76c dvc: support group cache mode 
In this mode, dvc will set appropriate file/dir permissions to support
sharing cache between different users within the same group.

Fixes #2287

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev 5aae88b316 local: don't try to protect file if it is already protected 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-09 16:46:24 +03:00
Ruslan Kuprieiev 4dd221dab9 reproduce: properly handle --single-item case 
Fix #2354

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-08-01 22:36:47 +03:00
Ruslan Kuprieiev 56b162d2d0 readme: fixed formatting 2019-07-31 21:09:46 +03:00
Ruslan Kuprieiev 3bdcaba39f travis: osx: retry brew update 
Fixes #2347

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-31 20:49:08 +03:00
Ruslan Kuprieiev 79052ddd5c readme: fix message about the number of install options 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-31 20:37:30 +03:00
Ruslan Kuprieiev 5e02a90e13 version: don't skip link info and skip if there is no psutil 
Fixes #2336

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-29 03:33:07 +03:00
Ruslan Kuprieiev 4a4ddf078c version: temporarily make psutil optional 
Workaround for https://github.com/iterative/dvc/issues/2284

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-25 01:52:11 +03:00
Ruslan Kuprieiev 641edacef6 import-url: lock by-default 
Fixes #2238

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-22 20:24:00 +03:00
Ruslan Kuprieiev 4a0b3a807d import: only donwload specific output 
Fixes #2252

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-22 15:04:10 +03:00
Ruslan Kuprieiev f901ba5c10 tests: use moto pre-release 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-20 16:09:44 +03:00
Ruslan Kuprieiev 68bebe7944 remote: hdfs: use pyarrow and connection pool 
Related to #1629

Speeds up HDFS tests significantly. E.g. ExternalHDFS test goes from 600
sec to 190sec. test_open_external[HDFS] from 160sec to 3sec.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-20 01:12:17 +03:00
Ruslan Kuprieiev 2013e21aaf checkout: safe_remove output if no checksum is found 
Fixes #2146

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-05 02:59:57 +03:00
Ruslan Kuprieiev da63c0ffdc travis: enable windows build and ditch appveyor 
Fixes #2176

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-03 21:09:23 +03:00
Ruslan Kuprieiev babc4a7f7e appveyor: require py3 to pass 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-25 04:48:51 +03:00
Ruslan Kuprieiev 1123a303bf import: require 'path' argument 
`dvc import` doesn't yet support importing the whole package.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-25 01:54:32 +03:00
Ruslan Kuprieiev 2141a5ac65 import: add "beta feature" warning 2019-06-25 01:54:32 +03:00
Ruslan Kuprieiev 12c7689a74 dvc: update paramiko requirements 
Fixes #2155

No longer need to explicitly ignore paramiko warnings.

Big thanks to @cowlinator for the heads-up!

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-24 00:24:06 +03:00
Ruslan Kuprieiev 9bca1c9bff dvc: use remove() instead of shutil.rmtree() 
Our `remove()` is able to handle read-only files which we or git
sometimes leave behind.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-23 18:11:30 +03:00
Ruslan Kuprieiev 918ef646dc appveyor: test py2 and py3 
Related to #2166

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-23 13:33:47 +03:00
Ruslan Kuprieiev e890c0558a dvc: fix help messages for get and import-url 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-22 00:50:22 +03:00
Ruslan Kuprieiev 81fd0b3415 dvc: destroy: don't remove data 
Fix #1011

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-21 13:13:29 +03:00
Ruslan Kuprieiev d795cdf1ba dvc: address review from #2160 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-21 12:27:24 +03:00
Ruslan Kuprieiev 1ba8e0171d dvc: replace pkg concept with external repo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-21 01:16:00 +03:00
Ruslan Kuprieiev be255ba4b0 http: enable progress bar 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-19 14:22:52 +03:00
Ruslan Kuprieiev d3bdca3f13 tests: use shortuuid for shorter paths 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-18 16:11:39 +03:00
Ruslan Kuprieiev 14dc0c0115 pkg: handle install/version errors 
Fixes #1772

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-18 16:09:06 +03:00
Ruslan Kuprieiev 4947069c4f pkg: introduce -f|--force for pkg install 
Fixes #1773

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-17 21:06:10 +03:00
Ruslan Kuprieiev 914a587123 pkg: implement dvc pkg get command 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-17 18:12:36 +03:00
Ruslan Kuprieiev 5941ace270 dvc: initial 'dvc pkg' implementation 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-17 18:12:36 +03:00
Ruslan Kuprieiev 0d330c1b9b dvc: move Updater out of Repo() 
Fixes #2039

Required by #2012 to be able to fetch data through the API without
interuption from updater.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-14 00:54:36 +03:00
Ruslan Kuprieiev b1eebe2f86 diff: don't use self 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-14 00:20:56 +03:00
Ruslan Kuprieiev bfdb990b3b utils: dont use str() for dvc_walk 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-13 11:51:14 +03:00
Ruslan Kuprieiev 39a8cab49d test: fix s3 test with extra args 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-11 16:28:11 +03:00
Ruslan Kuprieiev 8ef9133194 remote: use RemoteConfig class 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-06 16:00:44 +03:00
Ruslan Kuprieiev f69836c2b3 tests: remove fixme msg 
https://github.com/iterative/dvc/issues/2086

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-06 14:41:02 +03:00
Ruslan Kuprieiev 9e9e078de3 config: move logic from CLI to API 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-06 14:41:02 +03:00
Ruslan Kuprieiev 122b94129f Revert "tests: test hdfs on cron and releases" 
This reverts commit 76c4755865.
 2019-06-02 15:07:36 +03:00
Ruslan Kuprieiev 76c4755865 tests: test hdfs on cron and releases 
Related to
https://github.com/iterative/dvc/issues/1629#issuecomment-498023424

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-02 14:44:56 +03:00
Ruslan Kuprieiev 31848d5c7f remote: support remote:// notation for push/pull/gc 
Fixes #2066

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-29 12:54:31 +03:00
Ruslan Kuprieiev 4e9bc48f2a travis: build binary on python 3 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-25 02:40:16 +03:00
Ruslan Kuprieiev 095221ef7d appveyor: build windows binary on python 3.7 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-25 02:40:16 +03:00
Ruslan Kuprieiev abb6725a40 appveyor: adjust artifact regex 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 17:09:12 +03:00
Ruslan Kuprieiev 362866ba97 appveyor: specify artifact for specific provider 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 16:25:32 +03:00
Ruslan Kuprieiev 00bb09c3fe appveyor: don't deploy to pypi 
No need to deploy it again, since we have universal wheels anyways.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 15:36:17 +03:00
Ruslan Kuprieiev 5e49797264 travis: don't forget to install twine 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 15:26:16 +03:00
Ruslan Kuprieiev 7675888a83 appveyor: deploy to pypi 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 02:21:35 +03:00
Ruslan Kuprieiev b8fd41e923 travis: deploy to pypi 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 01:49:00 +03:00
Ruslan Kuprieiev 1d5d608c21 test: make sure checksums are not recomputed on checkout 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-13 21:07:12 +03:00
Ruslan Kuprieiev dd6217120f remote: checkout: save checksums without re-computation 
We do a similar thing on save, where we set cache checksum from already
computated ones for actual data. This patch does the same thing but in
revers: on checkout we know cache checksums, so we can set data
checksums from it without having to re-calculate it.

Fixes #1991

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-12 23:42:29 +03:00
Ruslan Kuprieiev b4143b19dd remote: unify class names 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-12 17:29:38 +03:00
Ruslan Kuprieiev a593a9ce2e dvc: refactor PathInfos 
Following same conventions from remotes/outsputs/dependencies.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-12 17:18:46 +03:00
Ruslan Kuprieiev d1881fd1b1 remote: separate http and https 
Fixes #1988

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-12 16:39:40 +03:00
Ruslan Kuprieiev 797b356e3a pyinstaller: remove hooks that are already released 
We've contributed these hooks a long time ago and they got finally
released, so we can remove our copies.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-11 21:44:55 +03:00
Ruslan Kuprieiev aacb0b4c98 pyinstaller: fix build for oss2 
This version added support for oss as a remote, which has caused some
issues when building the binary package. To fix those, we need two
pyinstaller hools:

1) hook-Crypto.py
    This one has been added to PyInstaller's upstream, but wasn't
    released yet, so we are using a copy of it for now.

2) hook-aliyunsdkcore.py
    This one is a new one. Sent it to PyInstaller's team in a PR [1].

[1] https://github.com/pyinstaller/pyinstaller/pull/4228

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-11 21:35:41 +03:00
Ruslan Kuprieiev d4ef461fce version: use CmdBaseNoRepo 
Currently `dvc version` outside of dvc repo raises "not dvc repo" error,
even though it is not really using repo in any way.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-11 17:20:30 +03:00
Ruslan Kuprieiev cc2f28c12b cmd: use CmdBaseNoRepo for commands without repo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-11 17:09:19 +03:00
Ruslan Kuprieiev 9c77f2818c stage: commit cached stage to avoid data duplication 
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 21:14:08 +03:00
Ruslan Kuprieiev 37ee55b1f8 test: test ssh 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 49f87a5bff use ValueError instead of JSONDecodeError 
JSONDecodeError is only available in never python 3 versions and still
inherits from ValueError, so we could just use it.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev b97b99cf15 test: func: use pytest for outputs/dependencies 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 44b95f23a3 remote: ssh: support directories 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 00cfb04fea remote: generalize file/dir handling 
Fixes #1654
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 41b67c3730 move: move directory into an existing directory 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 04:56:33 +03:00
Ruslan Kuprieiev 9618b32fe8 pkg: comply with design of other dvc commands 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-17 23:24:27 +03:00
Ruslan Kuprieiev 6f11f1f51f test: move func tests to tests/func 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-16 20:19:22 +03:00
Ruslan Kuprieiev 2460dc255e repo: scm_context: re-raise error directly 
This way we preserve original traceback.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-09 14:09:54 +03:00
Ruslan Kuprieiev 56b52cc9bf checkout/remove: properly handle broken symlinks 
Fixes #1856

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-09 14:09:54 +03:00
Ruslan Kuprieiev 54f7b43aac dvc: version: detect release 
See failed release at [1]. Previously we used to look at TRAVIS_TAG env
var, but now we will try to detect release by matching current tag with
base version.

[1] https://travis-ci.com/iterative/dvc/builds/107365596

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-07 22:55:20 +03:00
Ruslan Kuprieiev 1b6f51cafc version: unused pkg_dir argument 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-07 22:45:27 +03:00
Ruslan Kuprieiev 7c540da5c4 daemon: don't spawn recursively 
Fixes #1803

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-07 20:12:23 +03:00
Ruslan Kuprieiev e4ab3773dd pkg: move temp_git_repo to scm/git 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-06 23:29:18 +03:00
Ruslan Kuprieiev 5c008cfcb5 pkg: import git dynamically 
This allows dvc to work even if there is no git present, as long as user
is not using `dvc pkg` command directly.

Fixes #1835

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-06 23:29:18 +03:00
Ruslan Kuprieiev 8ef5976ac8 add: check if output is local when generating stage name 
When adding a remote file with

```
$ dvc add s3://bucket/path
```

it results in

```
Error: failed to add file - is local is not supported for s3
```

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-06 03:25:53 +03:00
Ruslan Kuprieiev 2699a47f3e travis: build binary packages only for specific python and os versions 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-04 20:33:39 +03:00
Ruslan Kuprieiev aeff967a13 dvc: exclude persist when computing md5 for the stage 
Fixes #1828

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-04 17:12:16 +03:00
Ruslan Kuprieiev 1c28214728 pyinstaller: hooks: fix google.cloud.storage hook 
Fixes #1697

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-03 18:50:15 +03:00
Ruslan Kuprieiev 6ec86e6694 pyinstaller: hook: remove checksumdir hook 
That module is not used anymore in dvc.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-03 18:47:28 +03:00
Ruslan Kuprieiev 3794f364d8 status: properly handle STATUS_MISSING case 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-31 21:04:19 +03:00
Ruslan Kuprieiev 0385dec5c0 dvc: use http remote for downloading dvc images 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-31 20:17:26 +03:00
Ruslan Kuprieiev 5109d759f1 system: reflink: workaround SIP error 
Fixes #1515

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-30 20:53:06 +02:00
Ruslan Kuprieiev 278f817ec7 hadoop: use --force when removing data 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 21:31:22 +02:00
Ruslan Kuprieiev 4c5cb7afae checkout: reorganize "git checkout hint" exception 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 13:44:32 +02:00
Ruslan Kuprieiev c03b1c1693 test: add tests for dvc tag 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 13:44:11 +02:00
Ruslan Kuprieiev 6e4d74282a dvc: temporarily hide dvc tag 
This feature is not yet ready for public use.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 13:44:11 +02:00
Ruslan Kuprieiev df971ec32d dvc: introduce dvc tag 
Fixes #1766

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 13:36:58 +02:00
Ruslan Kuprieiev 76c15ab80d dvc: repo: unify collect() method 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 12:43:19 +02:00
Ruslan Kuprieiev aaa9167a00 dvc: fix remote.checkout() to follow same conventions 
Currently `checkout()` accepts `OutputBase*` class as an argument, which
doesn't follow conventions from all other methods, which accept
path_info and/or checksum_info. To fix that, cache for dir info is moved
from output to the underlying remote.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 12:43:19 +02:00