Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-01 23:08:43 +01:00
Code Issues Packages Projects Releases Wiki Activity
3,094 Commits 7 Branches 579 Tags
5941ace270165cd4ab09402a83383cc525400349
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev 5941ace270 dvc: initial 'dvc pkg' implementation 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-17 18:12:36 +03:00
Ruslan Kuprieiev 0d330c1b9b dvc: move Updater out of Repo() 
Fixes #2039

Required by #2012 to be able to fetch data through the API without
interuption from updater.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-14 00:54:36 +03:00
Ruslan Kuprieiev b1eebe2f86 diff: don't use self 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-14 00:20:56 +03:00
Ruslan Kuprieiev bfdb990b3b utils: dont use str() for dvc_walk 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-13 11:51:14 +03:00
Ruslan Kuprieiev 39a8cab49d test: fix s3 test with extra args 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-11 16:28:11 +03:00
Ruslan Kuprieiev 8ef9133194 remote: use RemoteConfig class 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-06 16:00:44 +03:00
Ruslan Kuprieiev f69836c2b3 tests: remove fixme msg 
https://github.com/iterative/dvc/issues/2086

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-06 14:41:02 +03:00
Ruslan Kuprieiev 9e9e078de3 config: move logic from CLI to API 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-06 14:41:02 +03:00
Ruslan Kuprieiev 122b94129f Revert "tests: test hdfs on cron and releases" 
This reverts commit 76c4755865.
 2019-06-02 15:07:36 +03:00
Ruslan Kuprieiev 76c4755865 tests: test hdfs on cron and releases 
Related to
https://github.com/iterative/dvc/issues/1629#issuecomment-498023424

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-02 14:44:56 +03:00
Ruslan Kuprieiev 31848d5c7f remote: support remote:// notation for push/pull/gc 
Fixes #2066

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-29 12:54:31 +03:00
Ruslan Kuprieiev 4e9bc48f2a travis: build binary on python 3 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-25 02:40:16 +03:00
Ruslan Kuprieiev 095221ef7d appveyor: build windows binary on python 3.7 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-25 02:40:16 +03:00
Ruslan Kuprieiev abb6725a40 appveyor: adjust artifact regex 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 17:09:12 +03:00
Ruslan Kuprieiev 362866ba97 appveyor: specify artifact for specific provider 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 16:25:32 +03:00
Ruslan Kuprieiev 00bb09c3fe appveyor: don't deploy to pypi 
No need to deploy it again, since we have universal wheels anyways.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 15:36:17 +03:00
Ruslan Kuprieiev 5e49797264 travis: don't forget to install twine 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 15:26:16 +03:00
Ruslan Kuprieiev 7675888a83 appveyor: deploy to pypi 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 02:21:35 +03:00
Ruslan Kuprieiev b8fd41e923 travis: deploy to pypi 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-20 01:49:00 +03:00
Ruslan Kuprieiev 1d5d608c21 test: make sure checksums are not recomputed on checkout 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-13 21:07:12 +03:00
Ruslan Kuprieiev dd6217120f remote: checkout: save checksums without re-computation 
We do a similar thing on save, where we set cache checksum from already
computated ones for actual data. This patch does the same thing but in
revers: on checkout we know cache checksums, so we can set data
checksums from it without having to re-calculate it.

Fixes #1991

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-12 23:42:29 +03:00
Ruslan Kuprieiev b4143b19dd remote: unify class names 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-12 17:29:38 +03:00
Ruslan Kuprieiev a593a9ce2e dvc: refactor PathInfos 
Following same conventions from remotes/outsputs/dependencies.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-12 17:18:46 +03:00
Ruslan Kuprieiev d1881fd1b1 remote: separate http and https 
Fixes #1988

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-12 16:39:40 +03:00
Ruslan Kuprieiev 797b356e3a pyinstaller: remove hooks that are already released 
We've contributed these hooks a long time ago and they got finally
released, so we can remove our copies.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-11 21:44:55 +03:00
Ruslan Kuprieiev aacb0b4c98 pyinstaller: fix build for oss2 
This version added support for oss as a remote, which has caused some
issues when building the binary package. To fix those, we need two
pyinstaller hools:

1) hook-Crypto.py
    This one has been added to PyInstaller's upstream, but wasn't
    released yet, so we are using a copy of it for now.

2) hook-aliyunsdkcore.py
    This one is a new one. Sent it to PyInstaller's team in a PR [1].

[1] https://github.com/pyinstaller/pyinstaller/pull/4228

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-11 21:35:41 +03:00
Ruslan Kuprieiev d4ef461fce version: use CmdBaseNoRepo 
Currently `dvc version` outside of dvc repo raises "not dvc repo" error,
even though it is not really using repo in any way.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-11 17:20:30 +03:00
Ruslan Kuprieiev cc2f28c12b cmd: use CmdBaseNoRepo for commands without repo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-05-11 17:09:19 +03:00
Ruslan Kuprieiev 9c77f2818c stage: commit cached stage to avoid data duplication 
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 21:14:08 +03:00
Ruslan Kuprieiev 37ee55b1f8 test: test ssh 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 49f87a5bff use ValueError instead of JSONDecodeError 
JSONDecodeError is only available in never python 3 versions and still
inherits from ValueError, so we could just use it.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev b97b99cf15 test: func: use pytest for outputs/dependencies 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 44b95f23a3 remote: ssh: support directories 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 00cfb04fea remote: generalize file/dir handling 
Fixes #1654
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 41b67c3730 move: move directory into an existing directory 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-24 04:56:33 +03:00
Ruslan Kuprieiev 9618b32fe8 pkg: comply with design of other dvc commands 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-17 23:24:27 +03:00
Ruslan Kuprieiev 6f11f1f51f test: move func tests to tests/func 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-16 20:19:22 +03:00
Ruslan Kuprieiev 2460dc255e repo: scm_context: re-raise error directly 
This way we preserve original traceback.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-09 14:09:54 +03:00
Ruslan Kuprieiev 56b52cc9bf checkout/remove: properly handle broken symlinks 
Fixes #1856

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-09 14:09:54 +03:00
Ruslan Kuprieiev 54f7b43aac dvc: version: detect release 
See failed release at [1]. Previously we used to look at TRAVIS_TAG env
var, but now we will try to detect release by matching current tag with
base version.

[1] https://travis-ci.com/iterative/dvc/builds/107365596

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-07 22:55:20 +03:00
Ruslan Kuprieiev 1b6f51cafc version: unused pkg_dir argument 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-07 22:45:27 +03:00
Ruslan Kuprieiev 7c540da5c4 daemon: don't spawn recursively 
Fixes #1803

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-07 20:12:23 +03:00
Ruslan Kuprieiev e4ab3773dd pkg: move temp_git_repo to scm/git 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-06 23:29:18 +03:00
Ruslan Kuprieiev 5c008cfcb5 pkg: import git dynamically 
This allows dvc to work even if there is no git present, as long as user
is not using `dvc pkg` command directly.

Fixes #1835

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-06 23:29:18 +03:00
Ruslan Kuprieiev 8ef5976ac8 add: check if output is local when generating stage name 
When adding a remote file with

```
$ dvc add s3://bucket/path
```

it results in

```
Error: failed to add file - is local is not supported for s3
```

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-06 03:25:53 +03:00
Ruslan Kuprieiev 2699a47f3e travis: build binary packages only for specific python and os versions 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-04 20:33:39 +03:00
Ruslan Kuprieiev aeff967a13 dvc: exclude persist when computing md5 for the stage 
Fixes #1828

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-04 17:12:16 +03:00
Ruslan Kuprieiev 1c28214728 pyinstaller: hooks: fix google.cloud.storage hook 
Fixes #1697

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-03 18:50:15 +03:00
Ruslan Kuprieiev 6ec86e6694 pyinstaller: hook: remove checksumdir hook 
That module is not used anymore in dvc.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-04-03 18:47:28 +03:00
Ruslan Kuprieiev 3794f364d8 status: properly handle STATUS_MISSING case 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-31 21:04:19 +03:00
Ruslan Kuprieiev 0385dec5c0 dvc: use http remote for downloading dvc images 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-31 20:17:26 +03:00
Ruslan Kuprieiev 5109d759f1 system: reflink: workaround SIP error 
Fixes #1515

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-30 20:53:06 +02:00
Ruslan Kuprieiev 278f817ec7 hadoop: use --force when removing data 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 21:31:22 +02:00
Ruslan Kuprieiev 4c5cb7afae checkout: reorganize "git checkout hint" exception 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 13:44:32 +02:00
Ruslan Kuprieiev c03b1c1693 test: add tests for dvc tag 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 13:44:11 +02:00
Ruslan Kuprieiev 6e4d74282a dvc: temporarily hide dvc tag 
This feature is not yet ready for public use.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 13:44:11 +02:00
Ruslan Kuprieiev df971ec32d dvc: introduce dvc tag 
Fixes #1766

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 13:36:58 +02:00
Ruslan Kuprieiev 76c15ab80d dvc: repo: unify collect() method 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 12:43:19 +02:00
Ruslan Kuprieiev aaa9167a00 dvc: fix remote.checkout() to follow same conventions 
Currently `checkout()` accepts `OutputBase*` class as an argument, which
doesn't follow conventions from all other methods, which accept
path_info and/or checksum_info. To fix that, cache for dir info is moved
from output to the underlying remote.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-29 12:43:19 +02:00
Ruslan Kuprieiev ee51ccab4a test: reword --exec explanation 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-17 15:44:02 +02:00
Ruslan Kuprieiev 5fbf4f6db3 Revert "cli: status: remove unused code" 
This reverts commit ac27b4076a.

Fixes #1727
 2019-03-14 12:46:57 +02:00
Ruslan Kuprieiev ac27b4076a cli: status: remove unused code 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-03-14 01:25:15 +02:00
Ruslan Kuprieiev ef9f7da038 setup: fix flake8 errors 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-22 19:52:09 +02:00
Ruslan Kuprieiev 9ca00adde8 dvc: modify flake8 configuration 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-22 19:36:19 +02:00
Ruslan Kuprieiev 9961956c01 output/deps: raise proper exception when file/dir doesn't exist 
From discussion on discord
https://discordapp.com/channels/485586884165107732/485596304961962003/548083580052701205

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-22 14:59:01 +02:00
Ruslan Kuprieiev 967954a35f travis: try caching pyenv 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-21 19:48:05 +02:00
Ruslan Kuprieiev 5579fee604 travis: cache pyenv 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-21 19:41:45 +02:00
Ruslan Kuprieiev e5107b1da3 ssh: external-outs: check if cache exists before copying 
Fixes #1635

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-19 21:33:47 +02:00
Ruslan Kuprieiev 01a120f9fc dvc: introduce dvc commit 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-16 21:19:36 +02:00
Ruslan Kuprieiev 26c9fc1ba4 status: improve message verbosity 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-16 21:18:54 +02:00
Ruslan Kuprieiev 1f18119b74 repro: introduce --no-commit 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-16 21:18:54 +02:00
Ruslan Kuprieiev 99f4107d58 add: introduce --no-commit 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-16 21:18:54 +02:00
Ruslan Kuprieiev 3bc0b9728b run: introduce --no-commit 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-16 21:18:26 +02:00
Ruslan Kuprieiev 0eab7ed29e stage: split save() into save() + commit() 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-16 21:18:26 +02:00
Ruslan Kuprieiev bbd27527ff remote: RFC: remote.save() 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-16 21:18:26 +02:00
Ruslan Kuprieiev 0f024a9655 dvc: optimize imports 
Currently `dvc` takes ~0.7sec to finish. After this patch: 0.17sec.
Profiling has been done using:

    python -X importtime -m dvc 2> log.log
    tuna log.log

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-16 07:45:03 +02:00
Ruslan Kuprieiev a3099bca6a metrics: fix bad metric file case handling 
Also some refactoring got into this, sorry :(

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-14 00:15:18 +02:00
Ruslan Kuprieiev 91ed40fbfa updater: don't use io.open when working with json 
Otherwise we get this on python2:
```
======================================================================
ERROR: test_fetch (tests.unit.updater.TestUpdater)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/efiop/.pyenv/versions/2.7.15/envs/2.7.15-dvc/local/lib/
python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/efiop/git/dvc/tests/unit/updater.py", line 34, in test_fetch
    self.dvc.updater.fetch(detach=False)
  File "/home/efiop/git/dvc/dvc/updater.py", line 77, in fetch
    self._with_lock(self._get_latest_version, "fetching")
  File "/home/efiop/git/dvc/dvc/updater.py", line 39, in _with_lock
    func()
  File "/home/efiop/git/dvc/dvc/updater.py", line 91, in _get_latest_version
    json.dump(convert_to_unicode(info), fobj)
  File "/usr/lib/python2.7/json/__init__.py", line 190, in dump
    fp.write(chunk)
TypeError: write() argument 1 must be unicode, not str
-------------------- >> begin captured logging << --------------------
```

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-12 01:40:07 +02:00
Ruslan Kuprieiev c0674dc134 dvc: refactor requirements.txt 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-10 22:42:50 +02:00
Ruslan Kuprieiev 1eee7e3226 travis: enable cron builds 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-09 23:52:15 +02:00
Ruslan Kuprieiev 005239ade7 py2: cast env keys to bytes 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-09 06:59:21 +02:00
Ruslan Kuprieiev 40c84fe4ed py2: fix environ issue 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-09 03:59:45 +02:00
Ruslan Kuprieiev d3bb4f85df travis: don't forget to init pyenv 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-09 02:12:17 +02:00
Ruslan Kuprieiev cbce38074e travis: fix typo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-09 02:01:52 +02:00
Ruslan Kuprieiev 1bb101d250 travis: debug pyenv 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-09 01:40:58 +02:00
Ruslan Kuprieiev 1d272f156f config: handle run outside of dvc repo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-09 01:38:49 +02:00
Ruslan Kuprieiev 05bd4d4dc4 travis: use pyenv shell 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-08 22:20:41 +02:00
Ruslan Kuprieiev e22da1e957 travis: fix typo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-08 21:52:13 +02:00
Ruslan Kuprieiev 4f36ce35ad travis: activate pyenv 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-08 21:39:15 +02:00
Ruslan Kuprieiev 28061fb287 travis: don't define google creds file path 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-08 10:16:13 +02:00
Ruslan Kuprieiev 369c86bfde travis: fix env var definitions 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-08 08:39:17 +02:00
Ruslan Kuprieiev 95368b8f52 test: use "flaky" nose plugin for flaky tests 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-08 06:05:16 +02:00
Ruslan Kuprieiev 82b9d33674 test: ssh: create external output dir 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-08 04:19:44 +02:00
Ruslan Kuprieiev 9091ef2dc0 test: ssh: don't create dirs 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-08 03:51:21 +02:00
Ruslan Kuprieiev 515211a4dd travis: increase ulimit -u 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-08 02:44:25 +02:00
Ruslan Kuprieiev 53e40e32ed tests: use paramiko instead of ssh cmds 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-08 01:35:05 +02:00
Ruslan Kuprieiev 441fc3ad97 add: don't try to save cached stages 
Cached stages are already saved, so there is no need to try to save them
once again.

Fixes #1581

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-06 14:26:21 +02:00
Ruslan Kuprieiev 94a8054d29 test: gs: refresh credentials 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-06 01:46:23 +02:00
Ruslan Kuprieiev 99ef6081c1 test: use creds when testing external gs deps/outs 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-06 00:13:19 +02:00
Ruslan Kuprieiev f6535183a4 travis: gcloud: don't forget to authenticate 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-02-05 22:50:00 +02:00