Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-05 23:09:32 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,333 Commits 7 Branches 579 Tags
83d831735c8082e4d7eaa8bbeebe3978294ceeaa
Commit Graph
219 Commits
Author SHA1 Message Date
Ruslan Kuprieiev c733b8130b dvc: add dvc metrics add/show/remove and dvc run -M|--metrics-no-cache 
Fixes #656

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-16 16:01:32 +03:00
Ruslan Kuprieiev 4271ddf11c dvc introduce dvc import 
Fixes #705

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-15 18:37:50 +03:00
Ruslan Kuprieiev b074c49318 dvc: support external local deps 
Fixes #764

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-15 03:35:53 +03:00
Ruslan Kuprieiev b57f829b1b utils: read in binary mode when converting CRLF -> LF 
Fixes #773

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-14 23:37:10 +03:00
Ruslan Kuprieiev 78e5b166fe dvc: replace StageNotFoundError with NotDvcFileError 
Fixes #760

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-13 14:00:03 +03:00
Ruslan Kuprieiev 5902e6066c remove: add -o|--outs and -p|--purge options 
Fixes #761

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-13 13:15:40 +03:00
Ruslan Kuprieiev ff0d91737b dvc: initial hdfs support 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-10 05:32:16 +03:00
Ruslan Kuprieiev a734e07776 dvc: support deps/outputs that are relative to specific remote 
Fixes #748

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-09 02:03:02 +03:00
Ruslan Kuprieiev c136297b82 stage: try to specify user's shell when running commands 
Fixes #749

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-07 23:19:29 +03:00
Ruslan Kuprieiev 38c585a5f9 checkout: create dir even if it is empty 
Fixes #737

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-06 02:37:51 +03:00
Ruslan Kuprieiev 9389422f0b dvc: add --cwd for dvc repro 
Fixes #401

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-06 01:42:06 +03:00
Ruslan Kuprieiev dd610c04fb test: add test for s3/gs out/dep/cache 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-04 22:28:15 +03:00
Ruslan Kuprieiev 88397ef430 dvc: use Remote concept in existing code 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-04 22:28:14 +03:00
Ruslan Kuprieiev e0a42f1265 outs: support gs/s3 as an output without cache 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-04 22:28:14 +03:00
Ruslan Kuprieiev ac3d37e58c test: temporarily disable gs dependency test on CI 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-24 18:58:20 +03:00
Ruslan Kuprieiev 2bbefc91ef deps: introduce s3/gs dependencies 
E.g.:
dvc run -d gs://bucket/file -o file gsutil cp gs://bucket/file file
dvc run -d s3://bucket/file -o file aws s3 cp s3://bucket/file file

Etags are used to track deps in both cases.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-24 18:24:18 +03:00
Ruslan Kuprieiev 21bf52ee9d dvc: split output and dependency classes 
Preparation for the 'external deps' patches.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-23 00:01:04 +03:00
Ruslan Kuprieiev c819ac9292 dvc: add cache dir to gitignore only if it is not external 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-21 17:00:37 +03:00
Ruslan Kuprieiev c063ce9ff5 dvc: try to add to gitignore every time for backward compatibility 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-14 17:09:29 +03:00
Ruslan Kuprieiev c7fec74516 dvc: store list of links to cleanup later 
Fixes #685

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-13 21:02:54 +03:00
Ruslan Kuprieiev 6156b7e1b1 dvc: introduce 'dvc remote list' 
Fixes #690

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-05 14:17:35 +03:00
Ruslan Kuprieiev 9a88a412d0 dvc: reproduce stage if it has no dependencies and has cmd 
This allows as to define conditional stages in the dependency graph.

Fixes #646
Fixes #648

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-29 17:43:32 +03:00
Ruslan Kuprieiev 21fc712edf cache: remove unused code 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-28 21:50:49 +03:00
Ruslan Kuprieiev 0a049c232b cache: configurable cache dir and cache type 
From now on we allow specifying standalone cache dirs as well as
specifying linking method(copy, symlink, hardlink, reflink) for it.
If 'cache type' is not specified in the config, dvc will try all of
them starting from reflink all the way down to copy.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-28 21:50:49 +03:00
Ruslan Kuprieiev 52c08b2926 dvc: checkout single file 
Fixes #666

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-24 13:45:30 +03:00
Ruslan Kuprieiev 441098b35e dvc: introduce dvc install for installing git hooks 
Fixes #277
Fixes #657

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-24 10:47:22 +03:00
Ruslan Kuprieiev 5985fd3c96 metrics: don't forget to checkout() 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-20 11:04:05 +03:00
Ruslan Kuprieiev 1e7d548ca4 dvc: add dvc metrics 
Fixes #592

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-19 20:47:37 +03:00
Ruslan Kuprieiev a1460a6d92 init: properly handle dvc init in directory without scm 
Fixes #660
Fixes #662

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-18 09:27:55 +03:00
Ruslan Kuprieiev 29a44689c5 Merge pull request #658 from seqpipe/add-directory-with-slash 
Proper adding of 'directory/'
 2018-04-17 12:46:42 +03:00
Emil Gotsev 286153a609 Proper adding of 'directory/' 
* added a test for adding single level directory with forward slash in the end
 * normalizing the path that is passed to the add method
 2018-04-17 12:21:32 +03:00
Ruslan Kuprieiev 3748164ddb cloud: aws: migrate to boto3 
This allows us to abandon our own additional config parameters for
aws cloud(like region) and use the ones from .aws/config by default.
Plus, boto3 automatically switches to multipartupload if file is bigger
than a certain limit, allowing us to remove a lot of our own code.

Fixes #650
Fixes #636
Fixes #601

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-13 06:06:52 +03:00
Ruslan Kuprieiev 0df69a1046 tests: sleep(1) before modifying data files 
This will help simulate realistic behaviour on filesystems with poor mtime resolution.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-10 23:36:50 +03:00
Ruslan Kuprieiev 2dcfbad022 test: call nosetests directly 
Previous method of calling nose.main() didn't really work on windows.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-10 02:59:38 +03:00
Ruslan Kuprieiev c4d3aac787 test: don't forget to increse ulimit for opened files 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-10 01:56:06 +03:00
Ruslan Kuprieiev 8839e72f07 dvc: stop using checksumdir and use our own way of checksuming a dir 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-10 01:56:06 +03:00
Ruslan Kuprieiev 6dc9fbd08e dvc: cloud: use local state for cmp_checksum 
Fixes #633

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-10 01:56:06 +03:00
Ruslan Kuprieiev 828f00bf86 test: add ssh key in travis script 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-08 23:46:08 +03:00
Ruslan Kuprieiev 541429c26e tests: cloud: fix naming scheme 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-08 23:17:51 +03:00
Ruslan Kuprieiev c4fa14eae8 tests: ssh: fix typo 2018-04-08 22:52:37 +03:00
Ruslan Kuprieiev 52a88d6063 test: remove unittests.sh 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-08 22:26:18 +03:00
Ruslan Kuprieiev 36d924c4c2 test: add test for ssh cloud 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-08 22:26:18 +03:00
Ruslan Kuprieiev 9fdf949723 dvc: use DataCloud.supported() method for url testing 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-08 00:28:31 +03:00
Ruslan Kuprieiev 3988397a3d tests: data cloud: limit jobs to 2 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-07 05:23:58 +03:00
Ruslan Kuprieiev eef12da7b6 tests: fix typo 2018-04-07 05:14:57 +03:00
Ruslan Kuprieiev f5ef256447 test: don't forget to copy gcp credentials 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-07 05:09:45 +03:00
Ruslan Kuprieiev e8b5af6cd9 test: test remotes in CLI 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-07 02:34:00 +03:00
Ruslan Kuprieiev a948f5ca90 gs: don't require ProjectName 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-05 21:31:20 +03:00
Ruslan Kuprieiev fb4633072e dvc: introduce remotes 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-05 21:31:20 +03:00
Ruslan Kuprieiev 5a8aef28f5 config: empty config by default + format verification 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-05 01:51:52 +03:00