Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-10 23:10:42 +01:00
Code Issues Packages Projects Releases Wiki Activity
646 Commits 7 Branches 579 Tags
927d58cfad91e5b96009342eae2e39735dbd0228
Commit Graph
646 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev 927d58cfad dvc: introduce stage files instead of state files 
These describe single stage of pipeline and are allowed to
be placed anywhere in current or parent directories(up to
project's root) of the data file.

By default, 'dvc run' writes stage file to ./stage.dvc, but
it can be changed by using '-f' option(see functest/test-repro.py).

For now we're traversing project tree to find needed stage files for
specific data files. This is very wasteful and not optimal, but will
do for now, as this patch is aimed at stabilizing design of our repo
files.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-12-07 22:52:18 +02:00
Ruslan Kuprieiev 0b639a9c09 state: rework state format 
Dependencies are now specified in a more natural way:

deps:
  path/to/file1: <md5>
  path/to/file2: <md5>

And outputs also have md5 sum, that allows us to track
them by it and not by separate md5 field in state file,
which will allow us to remove strict data <-> data.state
relation in the future.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-12-07 01:58:13 +02:00
Ruslan Kuprieiev 2ee71f8687 Merge pull request #335 from efiop/master 
travis: manually specify macos image
 2017-12-06 19:03:06 +02:00
Ruslan Kuprieiev ebba1270c4 travis: manually specify mac os image 
Fixes #332

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-12-06 12:50:37 +02:00
Dmitry Petrov 7d872cf8e7 Merge pull request #337 from dataversioncontrol/git_file_state_file_fix 
Git file output fix
 2017-12-06 00:56:13 -08:00
Dmitry 075415f6bc Git file output fix 2017-12-06 00:33:59 -08:00
Ruslan Kuprieiev 141351db99 Merge pull request #333 from efiop/master 
State/CommandFile and repository design changes
 2017-12-05 21:35:26 +02:00
Ruslan Kuprieiev bbe5e46b5c dvc: remove CommandFile 
As we're moving to more user friendly single *.dvc file approach, separate
command files become uneeded.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-12-05 20:56:17 +02:00
Ruslan Kuprieiev e853fc76aa dvc: get rid of state dir and cache_state in favor of single .dvc 
From now on everything is stored in filename + .dvc path, except for local_state.
This allows us to be more explicit and help users see everything they need in relation
to version control of their own data files.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-12-05 20:33:21 +02:00
Dmitry Petrov 3322638b1b Merge pull request #319 from dataversioncontrol/new_design_fixes_2 
New design fixes 2
 2017-12-05 02:49:12 -08:00
Dmitry 4c388089af Rename -r to -g (--out-git). Close #322 2017-12-05 01:03:06 -08:00
Dmitry 7c49a195d8 Detailed message: md5 sum doesn\'t match 2017-12-05 00:51:01 -08:00
Dmitry 071c909962 Executor formatting 2017-11-28 22:51:40 -08:00
Dmitry 6a3fad1a4b Checkout: no warnings when file doesn't exist 2017-11-28 21:56:51 -08:00
Dmitry cd75631403 Replace a string hack to regexp 2017-11-28 19:14:04 -08:00
Dmitry 868c4d1c34 Python 2-3 unification: do not return iterators 2017-11-28 19:03:21 -08:00
Ruslan Kuprieiev 10f22142f5 Merge pull request #318 from efiop/master 
Misc small fixes
 2017-11-29 00:00:37 +02:00
Ruslan Kuprieiev a828fee479 requirements: add pyyaml 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-28 21:47:23 +02:00
Ruslan Kuprieiev f12402771b init: add *.local_state to gitignore 
Fixes #313

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-28 21:44:19 +02:00
Dmitry Petrov add5552c02 Merge pull request #317 from dataversioncontrol/new_design_fixes 
New design fixes
 2017-11-27 01:52:42 -08:00
Dmitry 2a980f6e57 Fix func tests: get rid of python3 lazy evaluation 2017-11-27 00:33:40 -08:00
Dmitry 4b3827255d Get rid of code duplication 2017-11-26 22:57:39 -08:00
Dmitry 86ad3cb360 It is okay if output file does not exist 2017-11-26 22:50:09 -08:00
Dmitry b022fa62fe Do not checkout files without cache_state 2017-11-26 22:47:47 -08:00
Dmitry 40660b5120 Refactoring: extract CommandFile class 2017-11-26 21:49:06 -08:00
Dmitry 50f736d9d9 dvc run: regular file output. Close #314 2017-11-26 21:40:55 -08:00
Dmitry e5fe6b3732 Ignore .coverage file 2017-11-26 18:09:01 -08:00
Ruslan Kuprieiev 5d8d6d2f55 Merge pull request #307 from efiop/master 
Func tests and RFC
 2017-11-27 03:17:43 +02:00
Ruslan Kuprieiev 99965d2cee functest: add partial repro test 
Fixes #259

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-27 02:58:37 +02:00
Ruslan Kuprieiev f035926891 functest: add simple gc test 
Fixes #274

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-27 00:41:02 +02:00
Ruslan Kuprieiev 3ed37fd0f8 data: check that data is not in .dvc 
Fixes #270

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-27 00:04:41 +02:00
Ruslan Kuprieiev f1ae4ad56c Merge pull request #306 from efiop/master 
Major misc changes(Cmdfile, repro, run, local_state etc)
 2017-11-26 20:27:43 +02:00
Ruslan Kuprieiev 32822d50bd data_item: generalize state files 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 16:10:37 +02:00
Ruslan Kuprieiev 7c320bbff3 pull/push: generalize CmdDataBase class 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 16:10:37 +02:00
Ruslan Kuprieiev d9e06c07b0 state: use yaml for everything 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 15:36:07 +02:00
Ruslan Kuprieiev 8b23a45a21 RFC: redo reproduce_deps 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 15:36:07 +02:00
Ruslan Kuprieiev ad873497cd RFC: redo repro's init 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:59:39 +02:00
Ruslan Kuprieiev da1c834608 RFC: move is_repro_required into separate function 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:54:49 +02:00
Ruslan Kuprieiev d706ad280c merge: just throw exception when can't load *state file 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:47:48 +02:00
Ruslan Kuprieiev 84cbdd8566 Revert "Generalize CmdDataBase class" 
This reverts commit e753d7fc79.

It is totally broken.
 2017-11-26 13:42:47 +02:00
Ruslan Kuprieiev be159e7798 dvc: fix misc dvc stuff 
Please don't merge pull requests that don't pass our checks.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:42:47 +02:00
Ruslan Kuprieiev 10c9449d32 travis: temporarily disable unittests 
We're in the process of stabilizing dvc and for now fixing unittests
requires too much time that will be wasted next time big change comes,
so lets disable it for now. We will fix it and enable it when preparing
next release.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:42:47 +02:00
Ruslan Kuprieiev 0f4d0e8861 functest: fix tests 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:42:47 +02:00
Ruslan Kuprieiev 5acdfcde51 run: use REMAINDER for command 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:42:47 +02:00
Ruslan Kuprieiev 170a6e0d3e merge: use command file 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:42:47 +02:00
Ruslan Kuprieiev e70ebb1e3d functest: use new 'dvc run' semantics in tutorial 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:42:47 +02:00
Ruslan Kuprieiev 7620e14c82 repro: state: rework state format and repro internals 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:42:46 +02:00
Ruslan Kuprieiev 3083604a9c dvc: remove 'import' and replace it with 'add' 
Allows us to unify state files and drop another level of
abstraction.

Fixes #304

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:38:23 +02:00
Ruslan Kuprieiev c5648a33ec merge: don't forget to checkout cache_state file 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:38:23 +02:00
Ruslan Kuprieiev 096c842063 repro: drop 'changed files' 
No longer needed, as we check for md5 match anyway.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-26 13:38:23 +02:00