Compare commits

..
188 Commits
Author SHA1 Message Date
Ruslan Kuprieiev c6811ae005 dvc: bump to 0.19.10 2018-10-12 04:22:26 +03:00
Ruslan Kuprieiev 6ad674ca88 Merge pull request #1213 from efiop/master
state: use UNSIGNED INTEGER for inodes
2018-10-12 04:22:03 +03:00
Ruslan Kuprieiev 45afd36d76 state: use UNSIGNED INTEGER for inodes
Fixes #1207

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-12 03:51:29 +03:00
Ruslan Kuprieiev 0aac01ab88 state: add debug msg
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-12 03:51:29 +03:00
Ruslan Kuprieiev ce3bcc4af9 Merge pull request #1211 from efiop/master
dagascii: import asciimatics only when needed
2018-10-12 00:11:06 +03:00
Ruslan Kuprieiev 06064061d6 dagascii: import asciimatics only when needed
Related to #1210

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-11 23:53:56 +03:00
Ruslan Kuprieiev cae6e712c6 dvc: bump to 0.19.9 2018-10-11 16:11:17 +03:00
Ruslan Kuprieiev 4b525fe1fc system: fix typo in _reflink_darwin 2018-10-11 16:05:27 +03:00
Ruslan Kuprieiev 13a2e4692d Merge pull request #1201 from efiop/master
repro: show which dependencies have changed
2018-10-11 00:10:39 +03:00
Ruslan Kuprieiev 1de87bc91c repro: show which dependencies have changed
Fixes #1164

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-10 23:40:26 +03:00
Ruslan Kuprieiev 82b10a8256 Merge pull request #1200 from efiop/master
import: add a note about supported schemes
2018-10-10 02:59:09 +03:00
Ruslan Kuprieiev a736082595 import: add a note about supported schemes
Fixes #1199

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-10 02:47:20 +03:00
Ruslan Kuprieiev 4888ab82a7 dvc: bump to 0.19.8 2018-10-09 21:01:17 +03:00
Ruslan Kuprieiev bc93cc0bf6 Merge pull request #1198 from efiop/master
hdfs: don't use close_fds on Windows
2018-10-09 21:00:55 +03:00
Ruslan Kuprieiev 971f3259f1 hdfs: don't use close_fds on Windows
Fixes #1197

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-09 20:42:18 +03:00
Ruslan Kuprieiev a4f9789aa4 travis: upload coverage on python 2 2018-10-08 03:30:02 +03:00
Ruslan Kuprieiev 584972ba0a travis: upload coverage on every build 2018-10-08 03:08:35 +03:00
Ruslan Kuprieiev b2b76b84d6 Merge pull request #1195 from efiop/master
dvc: stop using ntfsutils module
2018-10-07 18:07:51 +03:00
Ruslan Kuprieiev a6ea2f3db9 dvc: stop using ntfsutils module
Fixes #1162

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-07 17:49:29 +03:00
Ruslan Kuprieiev 881f188016 tests: run in parallel mode 2018-10-07 03:15:56 +03:00
Ruslan Kuprieiev c3f145591b Merge pull request #1193 from efiop/master
test: upload coverage only on CRON
2018-10-07 02:19:27 +03:00
Ruslan Kuprieiev 1a8842d8c6 test: upload coverage only on CRON
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-07 02:09:46 +03:00
Ruslan Kuprieiev 143a7f79a1 test: add version requirements for codecov and coverage 2018-10-06 23:24:55 +03:00
Ruslan Kuprieiev 7bff8c6f89 dagascii: don't try to cover tty code 2018-10-06 16:46:01 +03:00
Ruslan Kuprieiev 2daa694bf2 Merge pull request #1192 from efiop/master
dagascii: add more keys
2018-10-06 15:52:52 +03:00
Ruslan Kuprieiev 27a96ef814 dagascii: add more keys
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-06 15:41:36 +03:00
Ruslan Kuprieiev 315bbfde3b Merge pull request #1191 from efiop/master
show: use asciimatics to provide scrollable graph
2018-10-06 14:31:43 +03:00
Ruslan Kuprieiev 53b37bb266 show: use asciimatics to provide scrollable graph
Fixes #1177

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-06 01:04:00 +03:00
Ruslan Kuprieiev 005db9bb3a Merge pull request #1190 from efiop/master
dvc: add a note about dvc.org/support to error msgs
2018-10-05 22:24:20 +03:00
Ruslan Kuprieiev 7b36fd007c dvc: add a note about dvc.org/support to error msgs
Fixes #1152

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-05 21:26:55 +03:00
Ruslan Kuprieiev 2c9b08a89b Merge pull request #1189 from mroutis/update-networkx
Update networkx package (== 2.1, >= 2.1)
2018-10-05 03:07:54 +03:00
Ruslan Kuprieiev 66a962976e Merge pull request #1188 from mroutis/add-aur-repo
Add AUR package
2018-10-05 03:07:28 +03:00
Mr. Outis 4eff31dd5e Update README noticing that the AUR package is not officially maintained 2018-10-04 18:55:26 -05:00
Mr. Outis 377be75dd0 Update networkx package in setup.py 2018-10-04 18:44:10 -05:00
Ruslan Kuprieiev e015b9a167 readme: add a new line 2018-10-05 02:14:12 +03:00
Mr. Outis 2a7ec2d2df Update networkx package (== 2.1, >= 2.1)
The latest version is 2.2 (https://pypi.org/project/networkx).

Updating it to prevent collisions with some package managers already
using the latest one.
2018-10-04 18:08:27 -05:00
Mr. Outis 3d34f4e8c7 Add arch linux repository 2018-10-04 18:03:56 -05:00
Ruslan Kuprieiev c10bd344ea Merge pull request #1185 from efiop/master
stage: properly report StageFileFormatError
2018-10-04 13:30:11 +03:00
Ruslan Kuprieiev 6a8cd31ac9 stage: properly report StageFileFormatError
Fixes #1186

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-04 13:21:01 +03:00
Ruslan Kuprieiev 2c49d7d004 Merge pull request #1182 from efiop/master
gc: add a comfirmation prompt
2018-10-03 00:00:15 +03:00
Ruslan Kuprieiev 990bbe32c1 gc: add a comfirmation prompt
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-02 23:39:45 +03:00
Ruslan Kuprieiev f3668b0a4e cli: status: fix typo 2018-10-02 01:41:45 +03:00
Ruslan Kuprieiev 73eaa2a611 dvc: bump to 0.19.7 2018-10-01 21:35:42 +03:00
Ruslan Kuprieiev e63ae3d676 Merge pull request #1181 from jeepkd/fix-aws-env
s3: fix dvc not loading AWS settings from environment variables
2018-10-01 21:32:22 +03:00
Jeep Kiddee 9528ad6a1d Fix dvc not loading AWS settings from environment variables 2018-10-02 00:59:59 +07:00
Ruslan Kuprieiev aea92e0bdf Merge pull request #1180 from efiop/master
Misc test improvements
2018-10-01 18:26:42 +03:00
Ruslan Kuprieiev d1077c4457 test: improve coverage
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-01 18:11:17 +03:00
Ruslan Kuprieiev edf73dc52f test: skip hdfs test if can't connect
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-01 16:29:55 +03:00
Ruslan Kuprieiev b587f7833a dvc: bump to 0.19.6 2018-10-01 15:47:20 +03:00
Ruslan Kuprieiev 998918c690 Merge pull request #1179 from efiop/master
Misc fixes
2018-10-01 15:44:54 +03:00
Ruslan Kuprieiev 8aa53a5e71 dvc: introduce per-remote --jobs default
Fixes #1167

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-01 15:28:38 +03:00
Ruslan Kuprieiev cc2ebb2099 dvc: introduce -T|--all-tags option
Fixes #1150

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-01 12:13:17 +03:00
Ruslan Kuprieiev b9439f2e5b dvc: bump to 0.19.5 2018-09-29 23:06:51 +03:00
Ruslan Kuprieiev d112d2a829 Merge pull request #1175 from efiop/master
Misc fixes
2018-09-28 21:55:58 +03:00
Ruslan Kuprieiev f4fc3f47e4 show: don't use asciicanvas package
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 21:42:59 +03:00
Ruslan Kuprieiev 417ad8706e state: use state version
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 21:42:59 +03:00
Ruslan Kuprieiev 2e09dbb592 output: be more verbose
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 21:42:59 +03:00
Ruslan Kuprieiev 67d63a4a5b updater: check for updates every day
Our release schedule got really quick, so we can't afford to wait a week between checking for updates, because there might be new important versions with bugfixes released every day. Checking once per day should not cause too much inconvenience for users.
2018-09-28 18:20:29 +03:00
Ruslan Kuprieiev 8eb6fd5285 Merge pull request #1173 from efiop/master
repro: introduce -P|--all-pipelines
2018-09-28 13:58:48 +03:00
Ruslan Kuprieiev 669b2ec303 repro: introduce -P|--all-pipelines
Fixes #1165

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 13:33:23 +03:00
Ruslan Kuprieiev 04b3cd40e0 Merge pull request #1172 from efiop/master
s3: support env variables
2018-09-28 12:26:55 +03:00
Ruslan Kuprieiev 12d6292ed6 s3: support env variables
Fixes #1160

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 12:17:00 +03:00
Ruslan Kuprieiev 27edc4f027 dvc: bump to 0.19.4 2018-09-28 11:05:52 +03:00
Ruslan Kuprieiev 5b0abe61ab Merge pull request #1171 from efiop/master
move: use relpath intead of rel_path property
2018-09-28 10:25:09 +03:00
Ruslan Kuprieiev 1e1daeaae1 move: use relpath intead of rel_path property
Fixes #1170

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 10:06:51 +03:00
Ruslan Kuprieiev ece2b8ae0b dvc: bump to 0.19.3 2018-09-27 08:58:36 +03:00
Ruslan Kuprieiev fcea9dccc5 Merge pull request #1166 from efiop/master
dvc: remove unused module
2018-09-26 16:26:45 +03:00
Ruslan Kuprieiev 51b390146a dvc: remove unused module
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-26 15:34:26 +03:00
Ruslan Kuprieiev 1a6dedfe60 cli: add a note that remove --outs is default behavior. 2018-09-25 09:07:32 +03:00
Ruslan Kuprieiev 88f0913f91 Merge pull request #1161 from peter-schmidbauer/patch-1
dvc: project: Fix typos in warnings
2018-09-24 12:28:10 +03:00
Peter Schmidbauer b38112795a dvc: project: Fix typos in warnings 2018-09-24 09:30:07 +02:00
Ruslan Kuprieiev 363db9125f pull: add dummy call to try establishing the connection (#1159)
Fixes #1151

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-24 04:41:55 +03:00
pared 89e03c8606 pipeline show add --dot option (#1141)
* pipeline show add --dot option

* refactor common code, pydot in requirements

* code formatting

* code formatting
2018-09-24 02:50:41 +03:00
Ruslan Kuprieiev 7bed7213dc Merge pull request #1158 from efiop/master
osx: reflink: check if clib has clonefile()
2018-09-23 14:46:36 +03:00
Ruslan Kuprieiev 76c2f812f0 osx: reflink: check if clib has clonefile()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 14:39:09 +03:00
Ruslan Kuprieiev 8d151273c8 Merge pull request #1157 from efiop/master
reflink: osx: check that not older than High Sierra
2018-09-23 13:44:27 +03:00
Ruslan Kuprieiev 10e10f9535 reflink: osx: check that not older than High Sierra
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 13:29:18 +03:00
Ruslan Kuprieiev 3fd5603921 Merge pull request #1156 from efiop/master
dvc: use ctypes instead of cffi
2018-09-23 12:53:52 +03:00
Ruslan Kuprieiev ff63376d49 dvc: use ctypes instead of cffi
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 12:45:16 +03:00
Ruslan Kuprieiev 5071b5026d setup: exclude tests 2018-09-23 09:56:19 +03:00
Ruslan Kuprieiev c08804eddd dvc: bump to 0.19.2 2018-09-23 08:59:14 +03:00
Ruslan Kuprieiev cbc8cf23f9 Merge pull request #1155 from efiop/master
requirements: require newer google-cloud
2018-09-23 08:51:16 +03:00
Ruslan Kuprieiev 5279b1ae8f requirements: stop using obsoleted google-cloud
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 08:36:42 +03:00
Ruslan Kuprieiev 057663696a Merge pull request #1154 from efiop/master
setup: misc fixes
2018-09-23 06:23:46 +03:00
Ruslan Kuprieiev 8b513fb336 setup: misc fixes
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 06:06:01 +03:00
Ivan Shcheklein 04373e81e8 update README links 2018-09-22 10:30:44 -07:00
Ivan Shcheklein 2601766323 update README links, description 2018-09-22 10:28:42 -07:00
Ruslan Kuprieiev ad216acce4 travis: osx: modify CPPFLAGS and LDFLAGS 2018-09-22 14:53:30 +03:00
Ruslan Kuprieiev 5753944161 travis: osx: install openssl 2018-09-22 13:40:25 +03:00
Ruslan Kuprieiev b7ac347c06 travis: install: upgrade setuptools 2018-09-22 13:02:40 +03:00
Ruslan Kuprieiev beedab77f1 dvc: bump to 0.19.1 2018-09-22 12:27:59 +03:00
Ruslan Kuprieiev b1b0e3413d Merge pull request #1148 from efiop/master
Build osx packages on older versions
2018-09-22 12:24:57 +03:00
Ruslan Kuprieiev c4593cd369 travis: build on older xcode version
Fixes #1147

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-22 12:12:45 +03:00
Ruslan Kuprieiev 6a55d1502f dvc: avoid using pyreflink module
Fixes #1149

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-22 12:11:18 +03:00
Ruslan Kuprieiev e93a87f451 dvc: bump to 0.19.0 2018-09-21 11:09:00 +03:00
Ruslan Kuprieiev 3fff8d771d Merge pull request #1145 from efiop/master
dvc: properly join paths for push/pull progress bar
2018-09-21 09:31:54 +03:00
Ruslan Kuprieiev 69bb753980 dvc: properly join paths for push/pull progress bar
Fixes #1095

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-21 09:22:06 +03:00
Ruslan Kuprieiev 1d654d0444 Merge pull request #1143 from efiop/master
dvc: introduce -d|--with-deps option
2018-09-21 09:14:22 +03:00
Ruslan Kuprieiev 0618d9d0dc dvc: introduce -d|--with-deps option
Fixes #1077

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-21 08:45:09 +03:00
Ruslan Kuprieiev dea130cc61 Merge pull request #1142 from jeepkd/support-git-submodule
scm: support git submodule
2018-09-21 00:24:32 +03:00
Jeep Kiddee bbd5096de7 scm: support git submodule
Fixes #1093
2018-09-21 02:17:28 +07:00
Ruslan Kuprieiev 545ea1efa5 Merge pull request #1139 from efiop/master
scm: require abs path
2018-09-20 00:48:19 +03:00
Ruslan Kuprieiev e1841f07cb scm: require abs path
Fixes #1136

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-20 00:14:16 +03:00
Ruslan Kuprieiev 04efd1d6d3 Merge pull request #1138 from efiop/master
test: don't forget to check return code
2018-09-19 23:49:11 +03:00
Ruslan Kuprieiev 89aa2a957e test: don't forget to check return code
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-19 23:38:58 +03:00
Ruslan Kuprieiev 1da4546515 Merge pull request #1137 from efiop/master
local: use is_local property
2018-09-19 22:01:46 +03:00
Ruslan Kuprieiev 49c22cb139 local: use is_local property
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-19 21:40:00 +03:00
Ruslan Kuprieiev 4aa26a0a3a Merge pull request #1135 from efiop/master
ssh: properly join chunks on python3
2018-09-19 19:31:41 +03:00
Ruslan Kuprieiev 8d5675a92c ssh: properly join chunks on python3
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-19 18:58:36 +03:00
Ruslan Kuprieiev 2b1ea1e4b9 Merge pull request #1133 from efiop/master
repro: introduce -p|--pipeline option
2018-09-19 10:01:13 +03:00
Ruslan Kuprieiev 34faad3e96 repro: introduce -p|--pipeline option
Fixes #1107

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-19 09:35:06 +03:00
Ruslan Kuprieiev 9dab3f2e49 ssh: fix order 2018-09-19 07:19:02 +03:00
Ruslan Kuprieiev 72e8b6983b Merge pull request #1132 from efiop/master
ssh: add 'password' and 'ask_password' options
2018-09-19 06:52:16 +03:00
Ruslan Kuprieiev 682737e4a2 ssh: add 'password' and 'ask_password' options
Fixes #1124

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-19 05:50:06 +03:00
Ruslan Kuprieiev 8932254c12 Merge pull request #1131 from efiop/master
dvc: list changed files
2018-09-18 13:14:39 +03:00
Ruslan Kuprieiev 0c97e19d5c dvc: list files changed
Fixes #1130

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-18 12:37:55 +03:00
Ruslan Kuprieiev e698703969 Merge pull request #1129 from efiop/master
run: check that -f doesn't contain any subdirs
2018-09-18 07:59:04 +03:00
Ruslan Kuprieiev 571713484b run: check that -f doesn't contain any subdirs
Fixes #1110

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-18 07:46:28 +03:00
Ruslan Kuprieiev 65a9370281 Merge pull request #1128 from efiop/master
state: limit number of entries
2018-09-18 03:53:18 +03:00
Ruslan Kuprieiev 0043bd5de3 state: limit number of rows
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-18 03:40:41 +03:00
Ruslan Kuprieiev 33d04d0d59 state: count number of inserts and track size of the db
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-18 01:33:31 +03:00
Ruslan Kuprieiev 5e3d7c6c2b Merge pull request #1126 from efiop/master
state optimization
2018-09-17 10:59:32 +03:00
Ruslan Kuprieiev 59dcdb02ea state: optimize operations
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev 23d00a15b9 state: remove unused method
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev 2c3c0b08bd state: save timestamp
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev 75d7fc2206 state: merge State and LinkState classes
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev d704c41682 dvc: use INTEGER for inode
Fixes #1117

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev 428811bb71 Merge pull request #1122 from efiop/master
dvc: don't scan output directories for dvc files
2018-09-15 07:45:21 +03:00
Ruslan Kuprieiev 470825aec5 dvc: don't scan output directories for dvc files
`dvc status` for directory with 100K files sped up from ~6 sec to ~2
sec.

Fixes #1121

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-15 07:24:00 +03:00
Ruslan Kuprieiev 5fbfc135da Merge pull request #1120 from efiop/master
Misc enhancements
2018-09-15 05:08:54 +03:00
Ruslan Kuprieiev b4e62772b8 state: merge state and link state
Fixes #1091

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-15 04:54:26 +03:00
Ruslan Kuprieiev 912f0f72e5 run: require at least some arguments
Fix #1100

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-15 04:54:26 +03:00
Ruslan Kuprieiev 5730b1d088 Merge pull request #1118 from efiop/master
repro: don't check for missing dependencies when running with `--dry`
2018-09-14 00:33:45 +03:00
Ruslan Kuprieiev 5c762f316e repro: don't check for missing dependencies when running with --dry
Fixes #1113

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-14 00:23:28 +03:00
Ruslan Kuprieiev 7791dc8cfb Merge pull request #1116 from efiop/master
state: remove unused params
2018-09-13 18:24:09 +03:00
Ruslan Kuprieiev dc344056c5 state: remove unused params
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-13 18:01:23 +03:00
Ruslan Kuprieiev 87e82daad2 Merge pull request #1112 from efiop/master
state: use sqlite
2018-09-12 16:24:34 +03:00
Ruslan Kuprieiev f044a6090f state: use sqlite
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-12 15:52:34 +03:00
Ruslan Kuprieiev c68823e8d0 dvc: bump to 0.18.15 2018-09-10 15:57:24 +03:00
Ruslan Kuprieiev a17dc2504d Merge pull request #1106 from efiop/master
stage: check that deps exist before running the command
2018-09-09 23:25:04 +03:00
Ruslan Kuprieiev 537e7c75a0 stage: check that deps exist before running the command
Fixes #1101

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-09 22:59:19 +03:00
Ruslan Kuprieiev 76d6b37b1e Merge pull request #1105 from efiop/master
ssh: don't recv() before checking with recv_ready()
2018-09-09 18:59:31 +03:00
Ruslan Kuprieiev 8d13ba164e ssh: don't recv() before checking with recv_ready()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-09 18:48:57 +03:00
Ruslan Kuprieiev 559c097394 dvc: bump to 0.18.14 2018-09-08 21:39:20 +03:00
Ruslan Kuprieiev c97bfea90f Merge pull request #1103 from efiop/master
ssh fixes
2018-09-08 21:30:44 +03:00
Ruslan Kuprieiev abf395da99 ssh: properly poll output
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-08 21:14:50 +03:00
Ruslan Kuprieiev 0687c4e199 ssh: add timeout
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-08 16:17:08 +03:00
Ruslan Kuprieiev d1931e3dc0 readme: add discord link 2018-09-06 23:20:57 +03:00
Ruslan Kuprieiev 78c9d1edab dvc: bump to 0.18.13 2018-09-05 17:12:19 +03:00
Ruslan Kuprieiev 7ef8645465 push/pull/fetch: lower the thread multiplier
Fixes #1094
2018-09-05 17:11:45 +03:00
Ruslan Kuprieiev 208540c66d dvc: bump to 0.18.12 2018-09-04 19:30:28 +03:00
Ruslan Kuprieiev 216a6e5c1c Merge pull request #1089 from efiop/master
state: use json.dumps() instead of dump()
2018-09-04 19:28:54 +03:00
Ruslan Kuprieiev 8d8a94bcb7 state: use json.dumps() instead of dump()
This actually 10 times faster, because everything is stored in the
memory.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-04 19:09:25 +03:00
Ruslan Kuprieiev 1a18c4d3f1 Merge pull request #1088 from efiop/master
state: limit db size and add a file size limit
2018-09-04 18:53:32 +03:00
Ruslan Kuprieiev 211b957d17 state: limit db size and add a file size limit
Fixes #1087

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-04 18:24:19 +03:00
Ruslan Kuprieiev 4f6967a017 dvc: bump to 0.18.11 2018-09-03 00:21:23 +03:00
Ruslan Kuprieiev c1ac1b4c6e Merge pull request #1082 from efiop/master
state: don't forget to lock before dumping db
2018-09-03 00:06:11 +03:00
Ruslan Kuprieiev 022eea8214 state: don't forget to lock before dumping db
Related to #1081

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-02 23:47:50 +03:00
Ruslan Kuprieiev f1f1a97fbf Merge pull request #1080 from efiop/master
state: don't forget subdirs when computing effective mtime
2018-09-01 15:56:36 +03:00
Ruslan Kuprieiev a3aefe5f16 state: don't forget subdirs when computing effective mtime
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-01 15:41:42 +03:00
Ivan Shcheklein de3d8b528f remove Slack from README 2018-08-31 10:34:50 -07:00
Ruslan Kuprieiev 9c041a375f test: sleep to compensate for poor mtime resolution 2018-08-31 05:09:32 +03:00
Ruslan Kuprieiev 086155daa6 dvc: bump to 0.18.10 2018-08-31 04:16:52 +03:00
Ruslan Kuprieiev d47e6c39ab Merge pull request #1076 from efiop/master
remote: properly remove remote section
2018-08-31 04:15:42 +03:00
Ruslan Kuprieiev 249b60c030 remote: properly remove remote section
Fixes #1075

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-31 04:07:31 +03:00
Ruslan Kuprieiev 392cd006e3 tests: sleep() before creating a file 2018-08-30 17:27:05 +03:00
Ruslan Kuprieiev 2b0391629b readme: add slack channel 2018-08-30 13:41:16 +03:00
Ruslan Kuprieiev 3b66176f6b dvc: bump to 0.18.9 2018-08-30 00:51:45 +03:00
Ruslan Kuprieiev baa6bcd035 Merge pull request #1073 from efiop/master
run: add `-y|--yes` option to bypass prompts
2018-08-30 00:51:00 +03:00
Ruslan Kuprieiev b159ace6d7 run: add -y|--yes option to bypass prompts
Fixes #1072

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-30 00:38:28 +03:00
Ruslan Kuprieiev ffde4a6a7a Merge pull request #1071 from efiop/master
readme fixes
2018-08-29 20:53:49 +03:00
Ruslan Kuprieiev 9a95698cd9 travis: check readme formatting
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 20:34:28 +03:00
Ruslan Kuprieiev 221dbf8245 readme: fix formatting
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 20:22:25 +03:00
Ruslan Kuprieiev 3d15391c7a dvc: bump to 0.18.8 2018-08-29 19:32:55 +03:00
Ruslan Kuprieiev f6f2ebb83e Merge pull request #1070 from efiop/master
dvc: properly compute mtime of the directory
2018-08-29 19:32:29 +03:00
Ruslan Kuprieiev ac80d59704 dvc: properly compute mtime of the directory
Fixes #1067

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 19:17:50 +03:00
Ruslan Kuprieiev 3574c8954b dvc: update to 0.18.7 2018-08-29 16:01:28 +03:00
Ruslan Kuprieiev 0d78705227 Merge pull request #1069 from efiop/master
Misc
2018-08-29 16:00:45 +03:00
Ruslan Kuprieiev bf6ed76678 config: don't forget to convert port to int
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 15:50:25 +03:00
Ruslan Kuprieiev 076a2145db test: don't forget to sleep before modifying a file
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 14:54:27 +03:00
Ruslan Kuprieiev bb3cca6530 dvc: bump to 0.18.6 2018-08-29 14:02:31 +03:00
Ruslan Kuprieiev 165422a066 Merge pull request #1068 from efiop/master
misc fixes
2018-08-29 13:59:30 +03:00
Ruslan Kuprieiev 49baf38871 dvc: properly compute mtime for the directory
Fixes #1067

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 13:52:20 +03:00
Ruslan Kuprieiev 3461df1d8f config: properly merge with local config
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 13:52:20 +03:00
Ruslan Kuprieiev 03a3cf2f80 dvc: bump to 0.18.5 2018-08-28 13:05:36 +03:00
Ruslan Kuprieiev 94911965c4 Merge pull request #1066 from efiop/master
misc pipeline improvements
2018-08-28 12:45:06 +03:00
Ruslan Kuprieiev 55b3c47bbf pipeline: process edges and nodes separately
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-28 11:36:06 +03:00
Ruslan Kuprieiev 825da210cf dvc: use __str__ method for deps/outs
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-28 11:36:06 +03:00
52 changed files with 1937 additions and 572 deletions
+12 -7
View File
@@ -16,18 +16,24 @@ python:
# version, that causes fpm to crash.
matrix:
include:
# 3.7 needs Xenial image because of https://github.com/travis-ci/travis-ci/issues/9069
- python: "3.7"
dist: xenial
sudo: true
- if: type = cron OR tag is present
os: osx
osx_image: xcode9.3
osx_image: xcode7.3
language: ruby
rvm: 2.4.0
env:
- PYTHON_VER=2.7.10
- PYENV_ROOT="$HOME/.pyenv"
- PATH="$PYENV_ROOT/bin:$PATH"
- CPPFLAGS="-I/usr/local/opt/openssl/include $CPPFLAGS"
- LDFLAGS="-L/usr/local/opt/openssl/lib $LDFLAGS"
- if: type = cron OR tag is present
os: osx
osx_image: xcode9.3
osx_image: xcode7.3
language: ruby
rvm: 2.4.0
env:
@@ -36,23 +42,20 @@ matrix:
- PATH="$PYENV_ROOT/bin:$PATH"
- if: type = cron OR tag is present
os: osx
osx_image: xcode9.3
osx_image: xcode7.3
language: ruby
rvm: 2.4.0
env:
- PYTHON_VER=3.5.0
- PYENV_ROOT="$HOME/.pyenv"
- PATH="$PYENV_ROOT/bin:$PATH"
# 3.7 needs Xenial image because of https://github.com/travis-ci/travis-ci/issues/9069
- python: "3.7"
dist: xenial
sudo: true
before_install:
- bash ./scripts/ci/before_install.sh
- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
eval "$(pyenv init -)";
PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install --skip-existing --keep --verbose "$PYTHON_VER";
pyenv shell "$PYTHON_VER";
brew install openssl;
brew cask install google-cloud-sdk;
fi
- which python
@@ -78,6 +81,8 @@ install:
- bash ./scripts/ci/install.sh
- pip install flake8
before_script:
# stop the build if there are any readme formatting errors
- python setup.py checkdocs
# stop the build if there are any flake8 errors
- flake8 dvc
script:
+23 -7
View File
@@ -2,9 +2,12 @@
:target: https://dvc.org
:alt: DVC logo
`Website <https://dvc.org>`_ • `Docs <https://dvc.org/doc>`_ • `Discuss <https://discuss.dvc.org/>`_ •
`Blog <https://blog.dataversioncontrol.com/>`_ • `Twitter <https://twitter.com/iterativeai>`_ •
`Tutorial <https://dvc.org/doc/tutorial>`_ • `Mailing List <https://sweedom.us10.list-manage.com/subscribe/post?u=a08bf93caae4063c4e6a351f6&id=24c0ecc49a>`_
`Website <https://dvc.org>`_
• `Docs <https://dvc.org/doc>`_
• `Twitter <https://twitter.com/iterativeai>`_
• `Chat (Community & Support) <https://dvc.org/chat>`_
• `Tutorial <https://dvc.org/doc/tutorial>`_
• `Mailing List <https://sweedom.us10.list-manage.com/subscribe/post?u=a08bf93caae4063c4e6a351f6&id=24c0ecc49a>`_
.. image:: https://travis-ci.org/iterative/dvc.svg?branch=master
:target: https://travis-ci.org/iterative/dvc
@@ -24,10 +27,12 @@
|
**Data Science Version Control** or **DVC** is an **open-source** tool for data science projects.
With a simple and flexible Git-like architecture and interface it helps data scientists:
**Data Science Version Control** or **DVC** is an **open-source** tool for data science and
machine learning projects. With a simple and flexible Git-like architecture and interface it
helps data scientists:
#. manage code together with data files and models;
#. manage **machine learning models** - versioning, including data sets and transformations (scripts) that were
used to generate models;
#. make projects **reproducible**;
@@ -107,6 +112,17 @@ or:
$ brew cask install iterative/homebrew-dvc/dvc
AUR
---
*Unofficial package*, any inquiries regarding the AUR package,
`refer to the maintainer <https://github.com/mroutis/pkgbuilds>`_.
.. code-block:: bash
# https://aur.archlinux.org/packages/dvc
$ yay -S dvc
Related technologies
====================
@@ -161,7 +177,7 @@ details.
:alt: 7
Mailing List
=========
============
Want to stay up to date? Want to help improve DVC by participating in our ocassional polls? Subscribe to our `mailing list <https://sweedom.us10.list-manage.com/subscribe/post?u=a08bf93caae4063c4e6a351f6&id=24c0ecc49a>`_. No spam, really low traffic.
+4 -4
View File
@@ -7,7 +7,7 @@ import os
import warnings
VERSION_BASE = '0.18.4'
VERSION_BASE = '0.19.10'
__version__ = VERSION_BASE
PACKAGEPATH = os.path.abspath(os.path.dirname(__file__))
@@ -33,13 +33,13 @@ if os.path.exists(os.path.join(HOMEPATH, 'setup.py')):
with open(VERSIONPATH, 'w+') as fd:
fd.write('# AUTOGENERATED by dvc/__init__.py\n')
fd.write('version = "{}"\n'.format(__version__))
except Exception:
except Exception: # pragma: no cover
pass
else:
else: # pragma: no cover
# Remove version.py so that it doesn't get into the release
if os.path.exists(VERSIONPATH):
os.unlink(VERSIONPATH)
else:
else: # pragma: no cover
# dvc was installed with pip or something. Hopefully we have our
# auto-generated version.py to help us provide a true version
try:
+127 -6
View File
@@ -3,7 +3,6 @@ from __future__ import print_function
import os
import sys
import argparse
from multiprocessing import cpu_count
from dvc.command.init import CmdInit
from dvc.command.destroy import CmdDestroy
@@ -47,7 +46,7 @@ class DvcParser(argparse.ArgumentParser):
sys.exit(2)
class VersionAction(argparse.Action):
class VersionAction(argparse.Action): # pragma: no cover
def __call__(self, parser, namespace, values, option_string=None):
print(VERSION)
sys.exit(0)
@@ -161,7 +160,15 @@ def parse_args(argv=None):
help=IMPORT_HELP)
import_parser.add_argument(
'url',
help='URL.')
help="URL. Supported urls: "
"'/path/to/file', "
"'C:\\\\path\\to\\file', "
"'s3://bucket/path/to/file', "
"'gs://bucket/path/to/file', "
"'hdfs://example.com/path/to/file', "
"'ssh://example.com:/path/to/file', "
"'remote://myremote/path/to/file'(see "
"`dvc remote` commands). ")
import_parser.add_argument(
'out',
help='Output.')
@@ -235,6 +242,14 @@ def parse_args(argv=None):
default=False,
help="Only create stage file without actually "
"running it.")
run_parser.add_argument(
'-y',
'--yes',
action='store_true',
default=False,
help="Automatic 'yes' answer to all prompts. E.g. "
"when '.dvc' file exists and dvc asks if you "
"want to overwrite it.")
run_parser.add_argument(
'command',
nargs=argparse.REMAINDER,
@@ -249,7 +264,7 @@ def parse_args(argv=None):
'-j',
'--jobs',
type=int,
default=8*cpu_count(),
default=None,
help='Number of jobs to run simultaneously.')
parent_cache_parser.add_argument(
'--show-checksums',
@@ -279,6 +294,19 @@ def parse_args(argv=None):
action='store_true',
default=False,
help='Fetch cache for all branches.')
pull_parser.add_argument(
'-T',
'--all-tags',
action='store_true',
default=False,
help='Fetch cache for all tags.')
pull_parser.add_argument(
'-d',
'--with-deps',
action='store_true',
default=False,
help='Fetch cache for all dependencies of the '
'specified target.')
pull_parser.set_defaults(func=CmdDataPull)
# Push
@@ -298,6 +326,19 @@ def parse_args(argv=None):
action='store_true',
default=False,
help='Push cache for all branches.')
push_parser.add_argument(
'-T',
'--all-tags',
action='store_true',
default=False,
help='Push cache for all tags.')
push_parser.add_argument(
'-d',
'--with-deps',
action='store_true',
default=False,
help='Push cache for all dependencies of the '
'specified target.')
push_parser.set_defaults(func=CmdDataPush)
# Fetch
@@ -317,6 +358,19 @@ def parse_args(argv=None):
action='store_true',
default=False,
help='Fetch cache for all branches.')
fetch_parser.add_argument(
'-T',
'--all-tags',
action='store_true',
default=False,
help='Fetch cache for all tags.')
fetch_parser.add_argument(
'-d',
'--with-deps',
action='store_true',
default=False,
help='Fetch cache for all dependencies of the '
'specified target.')
fetch_parser.set_defaults(func=CmdDataFetch)
# Status
@@ -337,6 +391,28 @@ def parse_args(argv=None):
'-r',
'--remote',
help='Remote repository to compare local cache to.')
status_parser.add_argument(
'-a',
'--all-branches',
action='store_true',
default=False,
help='Show status of a local cache compared to a '
'remote repository for all branches.')
status_parser.add_argument(
'-T',
'--all-tags',
action='store_true',
default=False,
help='Show status of a local cache compared to a '
'remote repository for all tags.')
status_parser.add_argument(
'-d',
'--with-deps',
action='store_true',
default=False,
help='Show status of a local cache compared to a '
'remote repository for all dependencies of the '
'specified target.')
status_parser.set_defaults(func=CmdDataStatus)
# Repro
@@ -388,6 +464,19 @@ def parse_args(argv=None):
default=False,
help='Ask for confirmation before reproducing each '
'stage.')
repro_parser.add_argument(
'-p',
'--pipeline',
action='store_true',
default=False,
help='Reproduce the whole pipeline that the '
'specified stage file belongs to.')
repro_parser.add_argument(
'-P',
'--all-pipelines',
action='store_true',
default=False,
help='Reproduce all pipelines in the project.')
repro_parser.set_defaults(func=CmdRepro)
# Remove
@@ -403,7 +492,7 @@ def parse_args(argv=None):
'--outs',
action='store_true',
default=True,
help='Only remove DVC file outputs.')
help='Only remove DVC file outputs.(default)')
remove_parser_group.add_argument(
'-p',
'--purge',
@@ -444,6 +533,12 @@ def parse_args(argv=None):
action='store_true',
default=False,
help='Collect garbage for all branches.')
gc_parser.add_argument(
'-T',
'--all-tags',
action='store_true',
default=False,
help='Collect garbage for all tags.')
gc_parser.add_argument(
'-c',
'--cloud',
@@ -454,6 +549,12 @@ def parse_args(argv=None):
'-r',
'--remote',
help='Remote repository to collect garbage in.')
gc_parser.add_argument(
'-f',
'--force',
action='store_true',
default=False,
help='Force garbage collection.')
gc_parser.set_defaults(func=CmdGC)
# Config
@@ -637,6 +738,12 @@ def parse_args(argv=None):
action='store_true',
default=False,
help='Show metrics for all branches.')
metrics_show_parser.add_argument(
'-T',
'--all-tags',
action='store_true',
default=False,
help='Show metrics for all tags.')
metrics_show_parser.set_defaults(func=CmdMetricsShow)
METRICS_ADD_HELP = 'Add metrics.'
@@ -772,6 +879,10 @@ def parse_args(argv=None):
action='store_true',
default=False,
help='Output DAG as ASCII.')
pipeline_show_parser.add_argument(
'--dot',
help='Write DAG in .dot format.'
)
pipeline_show_parser.add_argument(
'targets',
nargs='*',
@@ -783,7 +894,9 @@ def parse_args(argv=None):
if (issubclass(args.func, CmdRepro)
or issubclass(args.func, CmdPipelineShow)) \
and hasattr(args, 'targets') \
and len(args.targets) == 0:
and len(args.targets) == 0 \
and not (hasattr(args, 'all_pipelines')
and args.all_pipelines): # pragma: no cover
if hasattr(args, 'cwd'):
cwd = args.cwd
else:
@@ -797,4 +910,12 @@ def parse_args(argv=None):
pipeline_show_parser.error(msg)
args.targets = ['Dvcfile']
if issubclass(args.func, CmdRun) \
and len(args.deps) == 0 \
and len(args.outs) == 0 \
and len(args.outs_no_cache) == 0 \
and len(args.command) == 0: # pragma: no cover
run_parser.error("Too few arguments. Specify at least one: "
"'-d', '-o', '-O', 'command'.")
return args
+9 -3
View File
@@ -23,7 +23,9 @@ class CmdDataPull(CmdDataBase):
jobs=self.args.jobs,
remote=self.args.remote,
show_checksums=self.args.show_checksums,
all_branches=self.args.all_branches)
all_branches=self.args.all_branches,
all_tags=self.args.all_tags,
with_deps=self.args.with_deps)
except Exception as exc:
self.project.logger.error('Failed to pull data from the cloud',
exc)
@@ -38,7 +40,9 @@ class CmdDataPush(CmdDataBase):
jobs=self.args.jobs,
remote=self.args.remote,
show_checksums=self.args.show_checksums,
all_branches=self.args.all_branches)
all_branches=self.args.all_branches,
all_tags=self.args.all_tags,
with_deps=self.args.with_deps)
except Exception as exc:
self.project.logger.error('Failed to push data to the cloud', exc)
return 1
@@ -52,7 +56,9 @@ class CmdDataFetch(CmdDataBase):
jobs=self.args.jobs,
remote=self.args.remote,
show_checksums=self.args.show_checksums,
all_branches=self.args.all_branches)
all_branches=self.args.all_branches,
all_tags=self.args.all_tags,
with_deps=self.args.with_deps)
except Exception as exc:
self.project.logger.error('Failed to fetch data from the cloud',
exc)
+2 -2
View File
@@ -1,5 +1,4 @@
from dvc.command.base import CmdBase
from dvc.prompt import prompt
from dvc.exceptions import DvcException
@@ -10,7 +9,8 @@ class CmdDestroy(CmdBase):
u'all data files, as well as cache in .dvc/cache.\n' \
u'Are you sure you want to continue?'
if not self.args.force and not prompt(msg, False):
if not self.args.force \
and not self.project.prompt.prompt(msg, False):
msg = u'Cannot destroy without a confirmation from the ' \
u'user. Use \'-f\' to force.'
raise DvcException(msg)
+17
View File
@@ -3,7 +3,24 @@ from dvc.command.base import CmdBase
class CmdGC(CmdBase):
def run(self):
msg = 'This will remove all cache except the cache that is used in '
if not self.args.all_branches and not self.args.all_tags:
msg += 'the current git branch.'
elif self.args.all_branches and not self.args.all_tags:
msg += 'all git branches.'
elif not self.args.all_branches and self.args.all_tags:
msg += 'all git tags.'
else:
msg += 'all git branches and all git tags.'
self.project.logger.warn(msg)
msg = 'Are you sure you want to proceed?'
if not self.args.force and not self.project.prompt.prompt(msg):
return 1
self.project.gc(all_branches=self.args.all_branches,
all_tags=self.args.all_tags,
cloud=self.args.cloud,
remote=self.args.remote)
return 0
+2 -1
View File
@@ -28,7 +28,8 @@ class CmdMetricsShow(CmdBase):
self.project.metrics_show(self.args.path,
typ=typ,
xpath=xpath,
all_branches=self.args.all_branches)
all_branches=self.args.all_branches,
all_tags=self.args.all_tags)
except DvcException as exc:
self.project.logger.error('Failed to show metrics', exc)
return 1
+40 -13
View File
@@ -20,11 +20,11 @@ class CmdPipelineShow(CmdBase):
self.project.logger.info(stages[n].cmd)
elif outs:
for out in stages[n].outs:
self.project.logger.info(out.rel_path)
self.project.logger.info(str(out))
else:
self.project.logger.info(n)
def _show_ascii(self, target, commands, outs):
def __build_graph(self, target, commands, outs):
import networkx
from dvc.stage import Stage
@@ -38,35 +38,57 @@ class CmdPipelineShow(CmdBase):
G = pipelines[0]
stages = networkx.get_node_attributes(G, 'stage')
nodes = set()
nodes = []
for n in G.nodes():
stage = stages[n]
if commands:
if stage.cmd is None:
continue
nodes.append(stage.cmd)
elif outs:
for out in stage.outs:
nodes.append(str(out))
else:
nodes.append(stage.relpath)
edges = []
for e in G.edges():
from_stage = stages[e[0]]
to_stage = stages[e[1]]
if commands:
nodes.add(from_stage.cmd)
if to_stage.cmd is None:
continue
nodes.add(to_stage.cmd)
edges.append((from_stage.cmd, to_stage.cmd))
elif outs:
for from_out in from_stage.outs:
for to_out in to_stage.outs:
nodes.add(from_out.rel_path)
nodes.add(to_out.rel_path)
edges.append((from_out.rel_path,
to_out.rel_path))
edges.append((str(from_out),
str(to_out)))
else:
nodes.add(from_stage.relpath)
nodes.add(to_stage.relpath)
edges.append((from_stage.relpath, to_stage.relpath))
nodes = list(nodes)
assert len(nodes) > 0
return nodes, edges
def _show_ascii(self, target, commands, outs):
nodes, edges = self.__build_graph(target, commands, outs)
if not nodes:
return
d = Dagascii(nodes, edges)
d.draw()
def __write_dot(self, target, commands, outs, filename):
import networkx
from networkx.drawing.nx_pydot import write_dot
_, edges = self.__build_graph(target, commands, outs)
edges = [edge[::-1] for edge in edges]
simple_g = networkx.DiGraph()
simple_g.add_edges_from(edges)
write_dot(simple_g, filename)
def run(self, unlock=False):
for target in self.args.targets:
try:
@@ -74,6 +96,11 @@ class CmdPipelineShow(CmdBase):
self._show_ascii(target,
self.args.commands,
self.args.outs)
elif self.args.dot:
self.__write_dot(target,
self.args.commands,
self.args.outs,
self.args.dot)
else:
self._show(target,
self.args.commands,
+29 -1
View File
@@ -1,4 +1,6 @@
import os
import re
import configobj
from dvc.config import Config
from dvc.command.config import CmdConfig
@@ -23,9 +25,35 @@ class CmdRemoteAdd(CmdConfig):
class CmdRemoteRemove(CmdConfig):
def _remove_default(self, config_file, remote):
path = os.path.join(os.path.dirname(self.config_file),
config_file)
config = configobj.ConfigObj(path)
core = config.get(Config.SECTION_CORE, None)
if core is None:
return
default = core.get(Config.SECTION_CORE_REMOTE, None)
if default is None:
return
if default == remote:
del config[Config.SECTION_CORE][Config.SECTION_CORE_REMOTE]
if len(config[Config.SECTION_CORE]) == 0:
del config[Config.SECTION_CORE]
config.write()
def run(self):
section = Config.SECTION_REMOTE_FMT.format(self.args.name)
return self.unset(section, Config.SECTION_REMOTE_URL)
ret = self.unset(section)
if ret != 0:
return ret
self._remove_default(Config.CONFIG, self.args.name)
self._remove_default(Config.CONFIG_LOCAL, self.args.name)
return 0
class CmdRemoteModify(CmdConfig):
+3 -1
View File
@@ -20,7 +20,9 @@ class CmdRepro(CmdBase):
recursive=recursive,
force=self.args.force,
dry=self.args.dry,
interactive=self.args.interactive)
interactive=self.args.interactive,
pipeline=self.args.pipeline,
all_pipelines=self.args.all_pipelines)
if len(stages) == 0:
self.project.logger.info(CmdDataStatus.UP_TO_DATE_MSG)
+3
View File
@@ -5,6 +5,9 @@ from dvc.exceptions import DvcException
class CmdRun(CmdBase):
def run(self):
try:
if self.args.yes:
self.project.prompt.default = True
self.project.run(cmd=' '.join(self.args.command),
outs=self.args.outs,
outs_no_cache=self.args.outs_no_cache,
+4 -1
View File
@@ -29,7 +29,10 @@ class CmdDataStatus(CmdDataBase):
jobs=self.args.jobs,
cloud=self.args.cloud,
show_checksums=self.args.show_checksums,
remote=self.args.remote)
remote=self.args.remote,
all_branches=self.args.all_branches,
all_tags=self.args.all_tags,
with_deps=self.args.with_deps)
if st:
self._show(st, indent)
else:
+36 -2
View File
@@ -45,6 +45,14 @@ def to_bool(val):
return val.lower() == 'true'
def is_whole(val):
return int(val) >= 0
def is_percent(val):
return int(val) >= 0 and int(val) <= 100
class Config(object):
CONFIG = 'config'
CONFIG_LOCAL = 'config.local'
@@ -134,6 +142,9 @@ class Config(object):
SECTION_REMOTE_USER = 'user'
SECTION_REMOTE_PORT = 'port'
SECTION_REMOTE_KEY_FILE = 'keyfile'
SECTION_REMOTE_TIMEOUT = 'timeout'
SECTION_REMOTE_PASSWORD = 'password'
SECTION_REMOTE_ASK_PASSWORD = 'ask_password'
SECTION_REMOTE_SCHEMA = {
SECTION_REMOTE_URL: And(supported_url, error="Unsupported URL"),
Optional(SECTION_AWS_REGION): str,
@@ -143,14 +154,26 @@ class Config(object):
Optional(SECTION_GCP_PROJECTNAME): str,
Optional(SECTION_CACHE_TYPE): SECTION_CACHE_TYPE_SCHEMA,
Optional(SECTION_REMOTE_USER): str,
Optional(SECTION_REMOTE_PORT): int,
Optional(SECTION_REMOTE_PORT): Use(int),
Optional(SECTION_REMOTE_KEY_FILE): str,
Optional(SECTION_REMOTE_TIMEOUT): Use(int),
Optional(SECTION_REMOTE_PASSWORD): str,
Optional(SECTION_REMOTE_ASK_PASSWORD): And(str, is_bool, Use(to_bool)),
}
SECTION_STATE = 'state'
SECTION_STATE_ROW_LIMIT = 'row_limit'
SECTION_STATE_ROW_CLEANUP_QUOTA = 'row_cleanup_quota'
SECTION_STATE_SCHEMA = {
Optional(SECTION_STATE_ROW_LIMIT): And(Use(int), is_whole),
Optional(SECTION_STATE_ROW_CLEANUP_QUOTA): And(Use(int), is_percent),
}
SCHEMA = {
Optional(SECTION_CORE, default={}): SECTION_CORE_SCHEMA,
Optional(Regex(SECTION_REMOTE_REGEX)): SECTION_REMOTE_SCHEMA,
Optional(SECTION_CACHE, default={}): SECTION_CACHE_SCHEMA,
Optional(SECTION_STATE, default={}): SECTION_STATE_SCHEMA,
# backward compatibility
Optional(SECTION_AWS, default={}): SECTION_AWS_SCHEMA,
@@ -171,7 +194,7 @@ class Config(object):
# need to convert our config to dict before passing it to
self._config = self._lower(self._config)
local = self._lower(local)
self._config.update(local)
self._config = self._merge(self._config, local)
self._config = Schema(self.SCHEMA).validate(self._config)
@@ -182,6 +205,17 @@ class Config(object):
except Exception as ex:
raise ConfigError(ex)
@staticmethod
def _merge(first, second):
res = {}
sections = list(first.keys()) + list(second.keys())
for section in sections:
f = first.get(section, {}).copy()
s = second.get(section, {}).copy()
f.update(s)
res[section] = f
return res
@staticmethod
def _lower(config):
new_config = {}
+171 -14
View File
@@ -1,13 +1,170 @@
import sys
import math
from asciicanvas.asciicanvas import AsciiCanvas
from asciicanvas.style import Style
import select
from grandalf.graphs import Vertex, Edge, Graph
from grandalf.layouts import SugiyamaLayout
from grandalf.routing import route_with_lines, EdgeViewer
class AsciiCanvas(object):
def __init__(self, cols, lines):
assert cols > 1
assert lines > 1
self.cols = cols
self.lines = lines
self.canvas = [[' '] * cols for l in range(lines)]
def draw(self):
if sys.stdout.isatty(): # pragma: no cover
from asciimatics.screen import Screen
Screen.wrapper(self._do_draw)
else:
for line in self.canvas:
print(''.join(line))
def _do_draw(self, screen): # pragma: no cover
from asciimatics.event import KeyboardEvent
offset_x = 0
offset_y = 0
smaxrow, smaxcol = screen.dimensions
assert smaxrow > 1
assert smaxcol > 1
smaxrow -= 1
smaxcol -= 1
if self.lines + 1 > smaxrow:
max_y = self.lines + 1 - smaxrow
else:
max_y = 0
if self.cols + 1 > smaxcol:
max_x = self.cols + 1 - smaxcol
else:
max_x = 0
while True:
for y in range(smaxrow + 1):
y_index = offset_y + y
line = []
for x in range(smaxcol + 1):
x_index = offset_x + x
if len(self.canvas) > y_index \
and len(self.canvas[y_index]) > x_index:
line.append(self.canvas[y_index][x_index])
else:
line.append(' ')
assert len(line) == (smaxcol + 1)
screen.print_at(''.join(line), 0, y)
screen.refresh()
# NOTE: get_event() doesn't block by itself,
# so we have to do the blocking ourselves.
select.select([sys.stdin], [], [], None)
event = screen.get_event()
if not isinstance(event, KeyboardEvent):
continue
k = event.key_code
if k == screen.KEY_DOWN or k == ord('s'):
offset_y += 1
elif k == screen.KEY_PAGE_DOWN or k == ord('S'):
offset_y += smaxrow
elif k == screen.KEY_UP or k == ord('w'):
offset_y -= 1
elif k == screen.KEY_PAGE_UP or k == ord('W'):
offset_y -= smaxrow
elif k == screen.KEY_RIGHT or k == ord('d'):
offset_x += 1
elif k == ord('D'):
offset_x += smaxcol
elif k == screen.KEY_LEFT or k == ord('a'):
offset_x -= 1
elif k == ord('A'):
offset_x -= smaxcol
elif k == ord('q') or k == ord('Q'):
break
if offset_y > max_y:
offset_y = max_y
elif offset_y < 0:
offset_y = 0
if offset_x > max_x:
offset_x = max_x
elif offset_x < 0:
offset_x = 0
def point(self, x, y, char):
assert x >= 0
assert x < self.cols
assert y >= 0
assert y < self.lines
self.canvas[y][x] = char
def line(self, x0, y0, x1, y1, char):
if x0 > x1:
x1, x0 = x0, x1
y1, y0 = y0, y1
dx = x1 - x0
dy = y1 - y0
if dx == 0 and dy == 0:
self.point(x0, y0, char)
elif abs(dx) >= abs(dy):
for x in range(x0, x1 + 1):
if dx == 0:
y = y0
else:
y = y0 + int(round((x - x0) * dy / float((dx))))
self.point(x, y, char)
elif y0 < y1:
for y in range(y0, y1 + 1):
if dy == 0:
x = x0
else:
x = x0 + int(round((y - y0) * dx / float((dy))))
self.point(x, y, char)
else:
for y in range(y1, y0 + 1):
if dy == 0:
x = x0
else:
x = x1 + int(round((y - y1) * dx / float((dy))))
self.point(x, y, char)
def text(self, x, y, text):
for i, c in enumerate(text):
self.point(x + i, y, c)
def box(self, x0, y0, w, h):
assert w > 1
assert h > 1
w -= 1
h -= 1
for x in range(x0, x0 + w):
self.point(x, y0, '-')
self.point(x, y0 + h, '-')
for y in range(y0, y0 + h):
self.point(x0, y, '|')
self.point(x0 + w, y, '|')
self.point(x0, y0, '+')
self.point(x0 + w, y0, '+')
self.point(x0, y0 + h, '+')
self.point(x0 + w, y0 + h, '+')
class Dagascii(object):
def __init__(self, vertexes, edges):
#
@@ -100,22 +257,22 @@ class Dagascii(object):
assert end_x >= 0
assert end_y >= 0
canvas.add_line(start_x, start_y,
end_x, end_y,
style=Style('*'))
canvas.line(start_x, start_y,
end_x, end_y,
'*')
for v in self.sug.g.sV:
# NOTE: moving boxes w/2 to the left
x = v.view.xy[0] - v.view.w/2.0
y = v.view.xy[1]
canvas.add_nine_patch_rect(int(round(x - minx)),
int(round(y - miny)),
v.view.w,
v.view.h)
canvas.box(int(round(x - minx)),
int(round(y - miny)),
v.view.w,
v.view.h)
canvas.add_text(int(round(x - minx)) + 1,
int(round(y - miny)) + 1,
v.data)
canvas.text(int(round(x - minx)) + 1,
int(round(y - miny)) + 1,
v.data)
canvas.print_out()
canvas.draw()
+4 -1
View File
@@ -31,6 +31,9 @@ class DependencyBase(object):
self.project = stage.project
self.path = path
def __str__(self):
return self.path
@classmethod
def match(cls, url):
return re.match(cls.REGEX, url)
@@ -51,7 +54,7 @@ class DependencyBase(object):
@property
def exists(self):
return self.remote.exists([self.path_info])
return self.remote.exists([self.path_info])[0]
def changed(self):
raise NotImplementedError
+12 -2
View File
@@ -38,6 +38,15 @@ class DependencyLOCAL(DependencyBase):
self.path_info = {'scheme': 'local',
'path': self.path}
def __str__(self):
return self.rel_path
@property
def is_local(self):
assert os.path.isabs(self.path)
assert os.path.isabs(self.project.root_dir)
return self.path.startswith(self.project.root_dir)
@property
def sep(self):
return os.sep
@@ -58,7 +67,8 @@ class DependencyLOCAL(DependencyBase):
if not self.exists:
raise self.DoesNotExistError(self.rel_path)
if not os.path.isfile(self.path) and not os.path.isdir(self.path):
if not os.path.isfile(self.path) \
and not os.path.isdir(self.path): # pragma: no cover
raise self.IsNotFileOrDirError(self.rel_path)
if (os.path.isfile(self.path) and os.path.getsize(self.path) == 0) or \
@@ -68,7 +78,7 @@ class DependencyLOCAL(DependencyBase):
self.info = self.remote.save_info(self.path_info)
def dumpd(self):
if self.path.startswith(self.stage.project.root_dir):
if self.is_local:
path = self.remote.unixpath(os.path.relpath(self.path,
self.stage.cwd))
else:
+10 -1
View File
@@ -8,7 +8,7 @@ class DvcException(Exception):
if cause:
try:
self.cause_tb = traceback.format_exc()
except Exception:
except Exception: # pragma: no cover
pass
super(DvcException, self).__init__(msg)
@@ -29,3 +29,12 @@ class OutputDuplicationError(DvcException):
for stage in stages:
s += "\n " + stage
super(OutputDuplicationError, self).__init__(msg.format(output, s))
class MoveNotDataSourceError(DvcException):
def __init__(self, path):
msg = "Move is not permitted for stages that are not data sources. " \
"You need to either move '{path}' to a new location and edit " \
"it by hand, or remove '{path}' and create a new one at the " \
"desired location."
super(MoveNotDataSourceError, self).__init__(msg.format(path=path))
+17 -7
View File
@@ -19,9 +19,16 @@ class Logger(object):
}
COLOR_MAP = {
'debug': colorama.Fore.BLUE,
'warn': colorama.Fore.YELLOW,
'error': colorama.Fore.RED
'green': colorama.Fore.GREEN,
'yellow': colorama.Fore.YELLOW,
'blue': colorama.Fore.BLUE,
'red': colorama.Fore.RED,
}
LEVEL_COLOR_MAP = {
'debug': 'blue',
'warn': 'yellow',
'error': 'red',
}
def __init__(self, loglevel=None):
@@ -58,12 +65,12 @@ class Logger(object):
Logger.logger().setLevel(logging.DEBUG)
@staticmethod
def colorize(msg, typ):
def colorize(msg, color):
header = ''
footer = ''
if sys.stdout.isatty(): # pragma: no cover
header = Logger.COLOR_MAP.get(typ.lower(), '')
header = Logger.COLOR_MAP.get(color.lower(), '')
footer = colorama.Style.RESET_ALL
return u'{}{}{}'.format(header, msg, footer)
@@ -84,7 +91,8 @@ class Logger(object):
@staticmethod
def _prefix(msg, typ):
return Logger.colorize('{}: '.format(msg), typ)
color = Logger.LEVEL_COLOR_MAP.get(typ.lower(), '')
return Logger.colorize('{}: '.format(msg), color)
@staticmethod
def error_prefix():
@@ -105,7 +113,9 @@ class Logger(object):
if Logger.logger().getEffectiveLevel() == logging.DEBUG and exc:
str_tb = str_tb if str_tb else traceback.format_exc()
Logger.logger().error(prefix + str_tb)
return Logger.logger().error(prefix + msg + str_exc)
chat = "\n\nHaving any troubles? Hit us up at dvc.org/support, " \
"we are always happy to help!"
return Logger.logger().error(prefix + msg + str_exc + chat)
@staticmethod
def warn(msg):
+4 -7
View File
@@ -40,10 +40,6 @@ class OutputLOCAL(DependencyLOCAL):
def cache(self):
return self.project.cache.local.get(self.md5)
@property
def is_local(self):
return self.path.startswith(self.project.root_dir)
def dumpd(self):
ret = super(OutputLOCAL, self).dumpd()
ret[self.PARAM_CACHE] = self.use_cache
@@ -87,13 +83,14 @@ class OutputLOCAL(DependencyLOCAL):
super(OutputLOCAL, self).save()
self._verify_metric()
msg = 'Output \'{}\' doesn\'t use cache. Skipping saving.'
self.project.logger.debug(msg.format(self.rel_path))
self.project.logger.info(msg.format(self.rel_path))
return
if not os.path.exists(self.path):
raise self.DoesNotExistError(self.rel_path)
if not os.path.isfile(self.path) and not os.path.isdir(self.path):
if not os.path.isfile(self.path) \
and not os.path.isdir(self.path): # pragma: no cover
raise self.IsNotFileOrDirError(self.rel_path)
if (os.path.isfile(self.path) and os.path.getsize(self.path) == 0) or \
@@ -103,7 +100,7 @@ class OutputLOCAL(DependencyLOCAL):
if not self.changed():
msg = 'Output \'{}\' didn\'t change. Skipping saving.'
self.project.logger.debug(msg.format(self.rel_path))
self.project.logger.info(msg.format(self.rel_path))
return
if self.is_local:
+5 -1
View File
@@ -16,6 +16,10 @@ class Progress(object):
self._n_total = total
self._n_finished = 0
@property
def is_finished(self):
return self._n_total == self._n_finished
def _clearln(self):
print('\r\x1b[K', end='')
@@ -63,7 +67,7 @@ class Progress(object):
progress = int((100 * current)/total) if current < total else 100
percent = str(progress) + "% "
if self._n_total:
if self._n_total > 1:
num = "({}/{}): ".format(self._n_finished + 1, self._n_total)
else:
num = ""
+288 -117
View File
@@ -1,6 +1,6 @@
import os
from dvc.exceptions import DvcException
from dvc.exceptions import DvcException, MoveNotDataSourceError
from dvc.stage import Stage
@@ -22,23 +22,23 @@ class Project(object):
def __init__(self, root_dir):
from dvc.logger import Logger
from dvc.config import Config
from dvc.state import LinkState, State
from dvc.state import State
from dvc.lock import Lock
from dvc.scm import SCM
from dvc.cache import Cache
from dvc.data_cloud import DataCloud
from dvc.updater import Updater
from dvc.prompt import Prompt
self.root_dir = os.path.abspath(os.path.realpath(root_dir))
self.dvc_dir = os.path.join(self.root_dir, self.DVC_DIR)
self.config = Config(self.dvc_dir)
self.scm = SCM(self.root_dir)
self.scm = SCM(self.root_dir, project=self)
self.lock = Lock(self.dvc_dir)
# NOTE: storing state and link_state in the repository itself to avoid
# any possible state corruption in 'shared cache dir' scenario.
self.state = State(self)
self.link_state = LinkState(self)
self.state = State(self, self.config._config)
core = self.config._config[Config.SECTION_CORE]
self.logger = Logger(core.get(Config.SECTION_CORE_LOGLEVEL, None))
@@ -46,11 +46,23 @@ class Project(object):
self.cache = Cache(self)
self.cloud = DataCloud(self, config=self.config._config)
self.updater = Updater(self.dvc_dir)
self.prompt = Prompt()
self._files_to_git_add = []
self._ignore()
self.updater.check()
def _remind_to_git_add(self):
if len(self._files_to_git_add) == 0:
return
msg = '\nTo track the changes with git run:\n\n'
msg += '\tgit add ' + " ".join(self._files_to_git_add)
self.logger.info(msg)
@staticmethod
def init(root_dir=os.curdir, no_scm=False, force=False):
"""
@@ -68,10 +80,10 @@ class Project(object):
import shutil
from dvc.scm import SCM, Base
from dvc.config import Config
from dvc.logger import Logger
root_dir = os.path.abspath(root_dir)
dvc_dir = os.path.join(root_dir, Project.DVC_DIR)
scm = SCM(root_dir)
if type(scm) == Base and not no_scm:
msg = "{} is not tracked by any supported scm tool(e.g. git)."
@@ -92,6 +104,8 @@ class Project(object):
if scm.ignore_file():
scm.add([os.path.join(dvc_dir, scm.ignore_file())])
Logger.info('You can now commit the changes to git.')
return proj
def destroy(self):
@@ -104,9 +118,6 @@ class Project(object):
def _ignore(self):
flist = [self.state.state_file,
self.state._lock_file.lock_file,
self.link_state.state_file,
self.link_state._lock_file.lock_file,
self.lock.lock_file,
self.config.config_local_file,
self.updater.updater_file]
@@ -150,16 +161,20 @@ class Project(object):
fnames = [fname]
stages = []
for f in fnames:
stage = Stage.loads(project=self,
outs=[f],
add=True)
self._files_to_git_add = []
with self.state:
for f in fnames:
stage = Stage.loads(project=self,
outs=[f],
add=True)
self._check_output_duplication(stage.outs)
self._check_output_duplication(stage.outs)
stage.save()
stage.dump()
stages.append(stage)
stage.save()
stage.dump()
stages.append(stage)
self._remind_to_git_add()
return stages
@@ -186,34 +201,37 @@ class Project(object):
[from_path])[0]
found = False
for stage in self.stages():
for out in stage.outs:
if out.path != from_out.path:
continue
self._files_to_git_add = []
with self.state:
for stage in self.stages():
for out in stage.outs:
if out.path != from_out.path:
continue
if not stage.is_data_source:
msg = 'Dvcfile \'{}\' is not a data source.'
raise DvcException(msg.format(stage.rel_path))
if not stage.is_data_source:
raise MoveNotDataSourceError(stage.relpath)
found = True
to_out = Output.loads_from(out.stage,
[to_path],
out.cache,
out.metric)[0]
out.move(to_out)
found = True
to_out = Output.loads_from(out.stage,
[to_path],
out.cache,
out.metric)[0]
out.move(to_out)
stage_base = os.path.basename(stage.path)
stage_base = stage_base.rstrip(Stage.STAGE_FILE_SUFFIX)
stage_base = os.path.basename(stage.path)
stage_base = stage_base.rstrip(Stage.STAGE_FILE_SUFFIX)
stage_dir = os.path.dirname(stage.path)
from_base = os.path.basename(from_path)
to_base = os.path.basename(to_path)
if stage_base == from_base:
os.unlink(stage.path)
path = to_base + Stage.STAGE_FILE_SUFFIX
stage.path = os.path.join(stage_dir, path)
stage_dir = os.path.dirname(stage.path)
from_base = os.path.basename(from_path)
to_base = os.path.basename(to_path)
if stage_base == from_base:
os.unlink(stage.path)
path = to_base + Stage.STAGE_FILE_SUFFIX
stage.path = os.path.join(stage_dir, path)
stage.dump()
stage.dump()
self._remind_to_git_add()
if not found:
msg = 'Unable to find dvcfile with output \'{}\''
@@ -227,7 +245,8 @@ class Project(object):
metrics_no_cache=[],
fname=Stage.STAGE_FILE,
cwd=os.curdir,
no_exec=False):
no_exec=False,
overwrite=False):
stage = Stage.loads(project=self,
fname=fname,
cmd=cmd,
@@ -236,13 +255,19 @@ class Project(object):
outs_no_cache=outs_no_cache,
metrics_no_cache=metrics_no_cache,
deps=deps,
overwrite=False)
overwrite=overwrite)
self._check_output_duplication(stage.outs)
if not no_exec:
stage.run()
self._files_to_git_add = []
with self.state:
if not no_exec:
stage.run()
stage.dump()
self._remind_to_git_add()
return stage
def imp(self, url, out):
@@ -253,15 +278,21 @@ class Project(object):
self._check_output_duplication(stage.outs)
stage.run()
self._files_to_git_add = []
with self.state:
stage.run()
stage.dump()
self._remind_to_git_add()
return stage
def _reproduce_stage(self, stages, node, force, dry, interactive):
stage = stages[node]
if stage.locked:
msg = 'DVC file \'{}\' is locked. Its dependecies are not ' \
msg = 'DVC file \'{}\' is locked. Its dependencies are not ' \
'going to be reproduced.'
self.logger.warn(msg.format(stage.relpath))
@@ -275,11 +306,60 @@ class Project(object):
return [stage]
def reproduce(self,
target,
target=None,
recursive=True,
force=False,
dry=False,
interactive=False):
interactive=False,
pipeline=False,
all_pipelines=False):
if target is None and not all_pipelines:
raise ValueError()
if not interactive:
config = self.config
core = config._config[config.SECTION_CORE]
interactive = core.get(config.SECTION_CORE_INTERACTIVE, False)
targets = []
if pipeline or all_pipelines:
if pipeline:
stage = Stage.load(self, target)
node = os.path.relpath(stage.path, self.root_dir)
pipelines = [self._get_pipeline(node)]
else:
pipelines = self.pipelines()
for G in pipelines:
for node in G.nodes():
if G.in_degree(node) == 0:
targets.append(os.path.join(self.root_dir, node))
else:
targets.append(target)
self._files_to_git_add = []
ret = []
with self.state:
for target in targets:
stages = self._reproduce(target,
recursive=recursive,
force=force,
dry=dry,
interactive=interactive)
ret.extend(stages)
self._remind_to_git_add()
return ret
def _reproduce(self,
target,
recursive=True,
force=False,
dry=False,
interactive=False):
import networkx as nx
stage = Stage.load(self, target)
@@ -287,20 +367,21 @@ class Project(object):
stages = nx.get_node_attributes(G, 'stage')
node = os.path.relpath(stage.path, self.root_dir)
if not interactive:
config = self.config
core = config._config[config.SECTION_CORE]
interactive = core.get(config.SECTION_CORE_INTERACTIVE, False)
if recursive:
return self._reproduce_stages(G,
stages,
node,
force,
dry,
interactive)
ret = self._reproduce_stages(G,
stages,
node,
force,
dry,
interactive)
else:
ret = self._reproduce_stage(stages,
node,
force,
dry,
interactive)
return self._reproduce_stage(stages, node, force, dry, interactive)
return ret
def _reproduce_stages(self, G, stages, node, force, dry, interactive):
import networkx as nx
@@ -322,7 +403,7 @@ class Project(object):
for stage in all_stages:
for out in stage.outs:
used.append(out.path)
self.link_state.remove_unused(used)
self.state.remove_unused_links(used)
def checkout(self, target=None):
all_stages = self.active_stages()
@@ -332,17 +413,46 @@ class Project(object):
else:
stages = all_stages
self._cleanup_unused_links(all_stages)
with self.state:
self._cleanup_unused_links(all_stages)
for stage in stages:
if stage.locked:
msg = 'DVC file \'{}\' is locked. Its dependecies are not ' \
'going to be checked out.'
self.logger.warn(msg.format(stage.relpath))
for stage in stages:
if stage.locked:
msg = 'DVC file \'{}\' is locked. Its dependencies are ' \
'not going to be checked out.'
self.logger.warn(msg.format(stage.relpath))
stage.checkout()
stage.checkout()
def _used_cache(self, target=None, all_branches=False, active=True):
def _get_pipeline(self, node):
pipelines = list(filter(lambda g: node in g.nodes(),
self.pipelines()))
assert len(pipelines) == 1
return pipelines[0]
def _collect(self, target, with_deps=False):
import networkx as nx
stage = Stage.load(self, target)
if not with_deps:
return [stage]
node = os.path.relpath(stage.path, self.root_dir)
G = self._get_pipeline(node)
stages = nx.get_node_attributes(G, 'stage')
ret = [stage]
for n in nx.dfs_postorder_nodes(G, node):
ret.append(stages[n])
return ret
def _used_cache(self,
target=None,
all_branches=False,
active=True,
with_deps=False,
all_tags=False):
cache = {}
cache['local'] = []
cache['s3'] = []
@@ -350,9 +460,11 @@ class Project(object):
cache['hdfs'] = []
cache['ssh'] = []
for branch in self.scm.brancher(all_branches=all_branches):
for branch in self.scm.brancher(all_branches=all_branches,
all_tags=all_tags):
if target:
stages = [Stage.load(self, target)]
stages = self._collect(target,
with_deps=with_deps)
elif active:
stages = self.active_stages()
else:
@@ -360,7 +472,7 @@ class Project(object):
for stage in stages:
if active and not target and stage.locked:
msg = 'DVC file \'{}\' is locked. Its dependecies are ' \
msg = 'DVC file \'{}\' is locked. Its dependencies are ' \
'not going to be pushed/pulled/fetched.'
self.logger.warn(msg.format(stage.relpath))
@@ -380,64 +492,90 @@ class Project(object):
if not removed:
self.logger.info("No unused {} cache to remove.".format(typ))
def gc(self, all_branches=False, cloud=False, remote=None):
clist = self._used_cache(target=None,
all_branches=all_branches,
active=False)
self._do_gc('local', self.cache.local.gc, clist)
def gc(self,
all_branches=False,
cloud=False,
remote=None,
with_deps=False,
all_tags=False):
with self.state:
clist = self._used_cache(target=None,
all_branches=all_branches,
active=False,
with_deps=with_deps,
all_tags=all_tags)
self._do_gc('local', self.cache.local.gc, clist)
if self.cache.s3:
self._do_gc('s3', self.cache.s3.gc, clist)
if self.cache.s3:
self._do_gc('s3', self.cache.s3.gc, clist)
if self.cache.gs:
self._do_gc('gs', self.cache.gs.gc, clist)
if self.cache.gs:
self._do_gc('gs', self.cache.gs.gc, clist)
if self.cache.ssh:
self._do_gc('ssh', self.cache.ssh.gc, clist)
if self.cache.ssh:
self._do_gc('ssh', self.cache.ssh.gc, clist)
if self.cache.hdfs:
self._do_gc('hdfs', self.cache.hdfs.gc, clist)
if self.cache.hdfs:
self._do_gc('hdfs', self.cache.hdfs.gc, clist)
if self.cache.azure:
self._do_gc('azure', self.cache.azure.gc, clist)
if self.cache.azure:
self._do_gc('azure', self.cache.azure.gc, clist)
if cloud:
self._do_gc('remote', self.cloud._get_cloud(remote,
'gc -c').gc, clist)
if cloud:
self._do_gc('remote', self.cloud._get_cloud(remote,
'gc -c').gc, clist)
def push(self,
target=None,
jobs=1,
remote=None,
all_branches=False,
show_checksums=False):
self.cloud.push(self._used_cache(target, all_branches)['local'],
jobs,
remote=remote,
show_checksums=show_checksums)
show_checksums=False,
with_deps=False,
all_tags=False):
with self.state:
used = self._used_cache(target,
all_branches=all_branches,
all_tags=all_tags,
with_deps=with_deps)['local']
self.cloud.push(used,
jobs,
remote=remote,
show_checksums=show_checksums)
def fetch(self,
target=None,
jobs=1,
remote=None,
all_branches=False,
show_checksums=False):
self.cloud.pull(self._used_cache(target, all_branches)['local'],
jobs,
remote=remote,
show_checksums=show_checksums)
show_checksums=False,
with_deps=False,
all_tags=False):
with self.state:
used = self._used_cache(target,
all_branches=all_branches,
all_tags=all_tags,
with_deps=with_deps)['local']
self.cloud.pull(used,
jobs,
remote=remote,
show_checksums=show_checksums)
def pull(self,
target=None,
jobs=1,
remote=None,
all_branches=False,
show_checksums=False):
show_checksums=False,
with_deps=False,
all_tags=False):
self.fetch(target,
jobs,
remote=remote,
all_branches=all_branches,
show_checksums=show_checksums)
all_tags=all_tags,
show_checksums=show_checksums,
with_deps=with_deps)
self.checkout(target=target)
def _local_status(self, target=None):
@@ -450,8 +588,8 @@ class Project(object):
for stage in stages:
if stage.locked:
msg = 'DVC file \'{}\' is locked. Its dependecies are not ' \
'going to be shown in status output.'
msg = 'DVC file \'{}\' is locked. Its dependencies are not ' \
'going to be shown in the status output.'
self.logger.warn(msg.format(stage.relpath))
status.update(stage.status())
@@ -462,11 +600,19 @@ class Project(object):
target=None,
jobs=1,
remote=None,
show_checksums=False):
show_checksums=False,
all_branches=False,
with_deps=False,
all_tags=False):
import dvc.remote.base as cloud
used = self._used_cache(target,
all_branches=all_branches,
all_tags=all_tags,
with_deps=with_deps)['local']
status = {}
for md5, ret in self.cloud.status(self._used_cache(target)['local'],
for md5, ret in self.cloud.status(used,
jobs,
remote=remote,
show_checksums=show_checksums):
@@ -487,13 +633,20 @@ class Project(object):
jobs=1,
cloud=False,
remote=None,
show_checksums=False):
if cloud:
return self._cloud_status(target,
jobs,
remote=remote,
show_checksums=show_checksums)
return self._local_status(target)
show_checksums=False,
all_branches=False,
with_deps=False,
all_tags=False):
with self.state:
if cloud:
return self._cloud_status(target,
jobs,
remote=remote,
show_checksums=show_checksums,
all_branches=all_branches,
with_deps=with_deps,
all_tags=all_tags)
return self._local_status(target)
def _read_metric_json(self, fd, json_path):
import json
@@ -573,9 +726,11 @@ class Project(object):
path=None,
typ=None,
xpath=None,
all_branches=False):
all_branches=False,
all_tags=False):
res = {}
for branch in self.scm.brancher(all_branches=all_branches):
for branch in self.scm.brancher(all_branches=all_branches,
all_tags=all_tags):
astages = self.active_stages()
outs = [out for stage in astages for out in stage.outs]
@@ -620,7 +775,7 @@ class Project(object):
res[branch][rel] = metric
for branch, val in res.items():
if all_branches:
if all_branches or all_tags:
self.logger.info('{}:'.format(branch))
for fname, metric in val.items():
self.logger.info('\t{}: {}'.format(fname, metric))
@@ -746,12 +901,28 @@ class Project(object):
def stages(self):
stages = []
outs = []
for root, dirs, files in os.walk(self.root_dir):
for fname in files:
path = os.path.join(root, fname)
if not Stage.is_stage_file(path):
continue
stages.append(Stage.load(self, path))
stage = Stage.load(self, path)
for out in stage.outs:
outs.append(out.path + out.sep)
stages.append(stage)
def filter_dirs(dname):
path = os.path.join(root, dname)
if path == self.dvc_dir or path == self.scm.dir:
return False
for out in outs:
if path == os.path.normpath(out) or path.startswith(out):
return False
return True
dirs[:] = list(filter(filter_dirs, dirs))
return stages
def active_stages(self):
+29 -7
View File
@@ -1,5 +1,7 @@
import sys
from dvc.progress import progress
try:
# NOTE: in Python3 raw_input() was renamed to input()
input = raw_input
@@ -7,12 +9,32 @@ except NameError:
pass
def prompt(msg, default=False): # pragma: no cover
if not sys.stdout.isatty():
return default
class Prompt(object):
def __init__(self):
self.default = None
answer = input(msg + u' (y/n)\n').lower()
while answer not in ['yes', 'no', 'y', 'n']:
answer = input('Enter \'yes\' or \'no\'.\n').lower()
def prompt(self, msg, default=False): # pragma: no cover
if self.default is not None:
return self.default
return answer[0] == "y"
if not sys.stdout.isatty():
return default
answer = input(msg + u' (y/n)\n').lower()
while answer not in ['yes', 'no', 'y', 'n']:
answer = input('Enter \'yes\' or \'no\'.\n').lower()
return answer[0] == "y"
def prompt_password(self, msg): # pragma: no cover
import getpass
if not sys.stdout.isatty():
return None
msg = 'Enter password for {}:\n'.format(msg)
if not progress.is_finished:
msg = u'\n' + msg
return getpass.getpass(msg)
+2
View File
@@ -1,6 +1,7 @@
import os
import re
import errno
from multiprocessing import cpu_count
from dvc.config import Config
from dvc.logger import Logger
@@ -30,6 +31,7 @@ class DataCloudError(DvcException):
class RemoteBase(object):
REGEX = None
REQUIRES = {}
JOBS = 4 * cpu_count()
def __init__(self, project, config):
pass
+6 -1
View File
@@ -29,9 +29,14 @@ class RemoteHDFS(RemoteBase):
cmd = 'hadoop fs -' + cmd
if user:
cmd = 'HADOOP_USER_NAME={} '.format(user) + cmd
# NOTE: close_fds doesn't work with redirected stdin/stdout/stderr.
# See https://github.com/iterative/dvc/issues/1197.
close_fds = (os.name != 'nt')
p = Popen(cmd,
shell=True,
close_fds=True,
close_fds=close_fds,
executable=os.getenv('SHELL'),
env=fix_env(os.environ),
stdin=PIPE,
+30 -26
View File
@@ -36,7 +36,6 @@ class RemoteLOCAL(RemoteBase):
def __init__(self, project, config):
self.project = project
self.state = self.project.state
self.link_state = project.link_state
storagepath = config.get(Config.SECTION_AWS_STORAGEPATH, None)
self.cache_dir = config.get(Config.SECTION_REMOTE_URL, storagepath)
@@ -109,7 +108,7 @@ class RemoteLOCAL(RemoteBase):
os.path.relpath(cache),
os.path.relpath(path)))
return
except Exception as exc:
except DvcException as exc:
msg = 'Cache type \'{}\' is not supported: {}'
Logger.debug(msg.format(self.cache_types[0], str(exc)))
del self.cache_types[0]
@@ -138,8 +137,8 @@ class RemoteLOCAL(RemoteBase):
def collect_dir_cache(self, dname):
dir_info = []
bar = False
for root, dirs, files in os.walk(dname):
bar = False
if len(files) > LARGE_DIR_SIZE:
msg = "Computing md5 for a large directory {}. " \
"This is only done once."
@@ -158,19 +157,12 @@ class RemoteLOCAL(RemoteBase):
progress.update_target(title, processed, total)
processed += 1
# FIXME: we could've used md5 = state.update(path, dump=False)
# here, but it is around twice as slow(on ssd, don't know about
# hdd) for a directory with small files. What we could do here
# is introduce some kind of a limit for file size, after which
# we would actually register it in our state file.
md5 = self.state.update(path, dump=False)
md5 = self.state.update(path)
dir_info.append({self.PARAM_RELPATH: relpath,
self.PARAM_MD5: md5})
self.state.dump()
if bar:
progress.finish_target(title)
if bar:
progress.finish_target(title)
# NOTE: sorting the list by path to ensure reproducibility
dir_info = sorted(dir_info, key=itemgetter(self.PARAM_RELPATH))
@@ -256,7 +248,7 @@ class RemoteLOCAL(RemoteBase):
if not self.is_dir_cache(cache):
self.link(cache, path)
self.link_state.update(path)
self.state.update_link(path)
return
# Create dir separately so that dir is created
@@ -270,7 +262,7 @@ class RemoteLOCAL(RemoteBase):
relpath = entry[self.PARAM_RELPATH]
p = os.path.join(path, relpath)
self.link(c, p)
self.link_state.update(path)
self.state.update_link(path)
def _move(self, inp, outp):
# moving in two stages to make last the move atomic in
@@ -292,7 +284,7 @@ class RemoteLOCAL(RemoteBase):
remove(path)
self.link(cache, path)
self.link_state.update(path)
self.state.update_link(path)
return {self.PARAM_MD5: md5}
@@ -313,7 +305,7 @@ class RemoteLOCAL(RemoteBase):
self.link(c, p)
self.link_state.update(path)
self.state.update_link(path)
return {self.PARAM_MD5: md5}
@@ -451,8 +443,8 @@ class RemoteLOCAL(RemoteBase):
for i in self.load_dir_cache(md5):
if info.get('branch'):
i['branch'] = info['branch']
i[self.PARAM_PATH] = posixpath.join(info[self.PARAM_PATH],
i[self.PARAM_RELPATH])
i[self.PARAM_PATH] = os.path.join(info[self.PARAM_PATH],
i[self.PARAM_RELPATH])
collected.append(i.copy())
collected.extend(checksum_infos)
@@ -495,11 +487,11 @@ class RemoteLOCAL(RemoteBase):
return removed
def status(self, checksum_infos, remote, jobs=1, show_checksums=False):
def status(self, checksum_infos, remote, jobs=None, show_checksums=False):
Logger.info("Preparing to pull data from {}".format(remote.url))
title = "Collecting information"
progress.set_n_total(0)
progress.set_n_total(1)
progress.update_target(title, 0, 100)
checksum_infos, missing = self._collect(checksum_infos)
@@ -531,11 +523,11 @@ class RemoteLOCAL(RemoteBase):
def _do_pull(self,
checksum_infos,
remote,
jobs=1,
jobs=None,
show_checksums=False):
title = "Collecting information"
progress.set_n_total(0)
progress.set_n_total(1)
progress.update_target(title, 0, 100)
grouped = zip(*self._group(checksum_infos,
@@ -561,8 +553,17 @@ class RemoteLOCAL(RemoteBase):
progress.update_target(title, 60, 100)
# NOTE: dummy call to try to establish a connection
# to see if we need to ask user for a password.
remote.exists(remote.md5s_to_path_infos(['000']))
progress.update_target(title, 70, 100)
assert len(path_infos) == len(cache) == len(md5s) == len(names)
if jobs is None:
jobs = remote.JOBS
chunks = list(zip(to_chunks(path_infos, jobs),
to_chunks(cache, jobs),
to_chunks(names, jobs)))
@@ -586,7 +587,7 @@ class RemoteLOCAL(RemoteBase):
for f in futures:
f.result()
def pull(self, checksum_infos, remote, jobs=1, show_checksums=False):
def pull(self, checksum_infos, remote, jobs=None, show_checksums=False):
Logger.info("Preparing to pull data from {}".format(remote.url))
# NOTE: try fetching missing dir info
@@ -603,11 +604,11 @@ class RemoteLOCAL(RemoteBase):
jobs,
show_checksums=show_checksums)
def push(self, checksum_infos, remote, jobs=1, show_checksums=False):
def push(self, checksum_infos, remote, jobs=None, show_checksums=False):
Logger.info("Preparing to push data to {}".format(remote.url))
title = "Collecting information"
progress.set_n_total(0)
progress.set_n_total(1)
progress.update_target(title, 0, 100)
checksum_infos = self._collect(checksum_infos)[0]
@@ -648,6 +649,9 @@ class RemoteLOCAL(RemoteBase):
progress.update_target(title, 90, 100)
if jobs is None:
jobs = remote.JOBS
chunks = list(zip(to_chunks(path_infos, jobs),
to_chunks(cache, jobs),
to_chunks(names, jobs)))
+12 -2
View File
@@ -47,7 +47,9 @@ class RemoteS3(RemoteBase):
'').lstrip('/')
self.url = config.get(Config.SECTION_REMOTE_URL, storagepath)
self.region = config.get(Config.SECTION_AWS_REGION, None)
self.profile = config.get(Config.SECTION_AWS_PROFILE, 'default')
self.profile = os.getenv('AWS_PROFILE',
config.get(Config.SECTION_AWS_PROFILE,
'default'))
self.endpoint_url = config.get(Config.SECTION_AWS_ENDPOINT_URL, None)
credentialpath = config.get(Config.SECTION_AWS_CREDENTIALPATH, None)
@@ -55,7 +57,7 @@ class RemoteS3(RemoteBase):
creds_conf = configobj.ConfigObj(credentialpath)
creds = creds_conf.get(self.profile, {})
else:
creds = {}
creds = self._get_aws_config_from_env()
self.creds = creds
self.region = creds.get('region', self.region)
@@ -81,6 +83,14 @@ class RemoteS3(RemoteBase):
region_name=self.region)
return session.client('s3', endpoint_url=self.endpoint_url)
def _get_aws_config_from_env(self):
creds = {
'region': os.getenv('AWS_DEFAULT_REGION'),
'aws_access_key_id': os.getenv('AWS_ACCESS_KEY_ID'),
'aws_secret_access_key': os.getenv('AWS_SECRET_ACCESS_KEY'),
}
return {k: v for k, v in creds.items() if v is not None}
def get_etag(self, bucket, key):
try:
obj = self.s3.head_object(Bucket=bucket, Key=key)
+55 -4
View File
@@ -44,9 +44,13 @@ class RemoteSSH(RemoteBase):
REGEX = r'^ssh://((?P<user>.*)@)?(?P<host>[^/]*):(?P<path>/.*)$'
REQUIRES = {'paramiko': paramiko}
JOBS = 4
PARAM_MD5 = 'md5'
DEFAULT_PORT = 22
TIMEOUT = 1800
def __init__(self, project, config):
self.project = project
@@ -59,6 +63,10 @@ class RemoteSSH(RemoteBase):
self.prefix = self.group('path')
self.port = config.get(Config.SECTION_REMOTE_PORT, self.DEFAULT_PORT)
self.keyfile = config.get(Config.SECTION_REMOTE_KEY_FILE, None)
self.timeout = config.get(Config.SECTION_REMOTE_TIMEOUT, self.TIMEOUT)
self.password = config.get(Config.SECTION_REMOTE_PASSWORD, None)
self.ask_password = config.get(Config.SECTION_REMOTE_ASK_PASSWORD,
False)
def md5s_to_path_infos(self, md5s):
return [{'scheme': 'ssh',
@@ -78,10 +86,16 @@ class RemoteSSH(RemoteBase):
ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
if self.ask_password and self.password is None:
msg = "host '{}' port '{}' user '{}'".format(host, port, user)
self.password = self.project.prompt.prompt_password(msg)
ssh.connect(host,
username=user,
port=port,
key_filename=self.keyfile)
key_filename=self.keyfile,
timeout=self.timeout,
password=self.password)
return ssh
@@ -105,10 +119,47 @@ class RemoteSSH(RemoteBase):
def _exec(self, ssh, cmd):
stdin, stdout, stderr = ssh.exec_command(cmd)
channel = stdout.channel
stdin.close()
channel.shutdown_write()
stdout_chunks = []
stderr_chunks = []
while (not channel.closed
or channel.recv_ready()
or channel.recv_stderr_ready()):
import select
got_chunk = False
readq, _, _ = select.select([stdout.channel], [], [], self.timeout)
for c in readq:
if c.recv_ready():
stdout_chunks.append(stdout.channel.recv(len(c.in_buffer)))
got_chunk = True
if c.recv_stderr_ready():
stderr_len = len(c.in_stderr_buffer)
s = stderr.channel.recv_stderr(stderr_len)
stderr_chunks.append(s)
got_chunk = True
if not got_chunk \
and stdout.channel.exit_status_ready() \
and not stderr.channel.recv_stderr_ready() \
and not stdout.channel.recv_ready():
stdout.channel.shutdown_read()
stdout.channel.close()
break
stdout.close()
stderr.close()
if stdout.channel.recv_exit_status() != 0:
DvcException('SSH command \'{}\' failed: {}'.format(cmd,
stderr.read()))
return stdout.read().decode('utf-8')
err = b''.join(stderr_chunks).decode('utf-8')
msg = 'SSH command \'{}\' failed: {}'.format(cmd, err)
raise DvcException(msg)
return b''.join(stdout_chunks).decode('utf-8')
def md5(self, path_info):
if path_info['scheme'] != 'ssh':
+68 -22
View File
@@ -14,13 +14,18 @@ class FileNotInRepoError(DvcException):
class Base(object):
def __init__(self, root_dir=os.curdir):
self.root_dir = os.path.abspath(os.path.realpath(root_dir))
def __init__(self, root_dir=os.curdir, project=None):
self.project = project
self.root_dir = root_dir
@staticmethod
def is_repo(root_dir):
return True
@staticmethod
def is_submodule(root_dir):
return True
def ignore(self, path):
pass
@@ -45,17 +50,34 @@ class Base(object):
def branch(self, branch):
pass
def brancher(self, branches=None, all_branches=False):
if not branches and not all_branches:
def brancher(self,
branches=None,
all_branches=False,
tags=None,
all_tags=False):
if not branches and not all_branches \
and not tags and not all_tags:
yield ''
return
saved = self.active_branch()
if not branches:
branches = self.list_branches() if all_branches else [saved]
for branch in branches:
self.checkout(branch)
yield branch
revs = []
if branches is not None:
revs.extend(branches)
elif all_branches:
revs.extend(self.list_branches())
elif tags is not None:
revs.extend(tags)
elif all_tags:
revs.extend(self.list_tags())
else:
revs.extend([saved])
for rev in revs:
self.checkout(rev)
yield rev
self.checkout(saved)
def untracked_files(self):
@@ -70,6 +92,9 @@ class Base(object):
def list_branches(self):
pass
def list_tags(self):
pass
def install(self):
pass
@@ -78,15 +103,16 @@ class Git(Base):
GITIGNORE = '.gitignore'
GIT_DIR = '.git'
def __init__(self, root_dir=os.curdir):
super(Git, self).__init__(root_dir)
def __init__(self, root_dir=os.curdir, project=None):
super(Git, self).__init__(root_dir, project=project)
import git
from git.exc import InvalidGitRepositoryError
try:
self.repo = git.Repo(root_dir)
except InvalidGitRepositoryError:
raise SCMError('{} is not a git repository'.format(root_dir))
msg = '{} is not a git repository'
raise SCMError(msg.format(root_dir))
# NOTE: fixing LD_LIBRARY_PATH for binary built by PyInstaller.
# http://pyinstaller.readthedocs.io/en/stable/runtime-information.html
@@ -96,17 +122,27 @@ class Git(Base):
@staticmethod
def is_repo(root_dir):
git_dir = os.path.join(root_dir, Git.GIT_DIR)
return os.path.isdir(git_dir)
return os.path.isdir(Git._get_git_dir(root_dir))
@staticmethod
def is_submodule(root_dir):
return os.path.isfile(Git._get_git_dir(root_dir))
@staticmethod
def _get_git_dir(root_dir):
return os.path.join(root_dir, Git.GIT_DIR)
@property
def dir(self):
return self.repo.git_dir
def ignore_file(self):
return self.GITIGNORE
def _get_gitignore(self, path):
assert os.path.isabs(path)
entry = os.path.basename(path)
gitignore = os.path.join(os.path.abspath(os.path.dirname(path)),
self.GITIGNORE)
gitignore = os.path.abspath(os.path.realpath(gitignore))
gitignore = os.path.join(os.path.dirname(path), self.GITIGNORE)
if not gitignore.startswith(self.root_dir):
raise FileNotInRepoError(path)
@@ -132,7 +168,11 @@ class Git(Base):
if len(ignore_list) > 0:
content = '\n' + content
open(gitignore, 'a').write(content)
with open(gitignore, 'a') as fd:
fd.write(content)
if self.project is not None:
self.project._files_to_git_add.append(os.path.relpath(gitignore))
def ignore_remove(self, path):
entry, gitignore = self._get_gitignore(path)
@@ -148,6 +188,9 @@ class Git(Base):
with open(gitignore, 'w') as fd:
fd.writelines(filtered)
if self.project is not None:
self.project._files_to_git_add.append(os.path.relpath(gitignore))
def add(self, paths):
# NOTE: GitPython is not currently able to handle index version >= 3.
# See https://github.com/iterative/dvc/issues/610 for more details.
@@ -185,6 +228,9 @@ class Git(Base):
def list_branches(self):
return [h.name for h in self.repo.heads]
def list_tags(self):
return [t.name for t in self.repo.tags]
def install(self):
hook = os.path.join(self.root_dir,
self.GIT_DIR,
@@ -198,8 +244,8 @@ class Git(Base):
os.chmod(hook, 0o777)
def SCM(root_dir=os.curdir, no_scm=False):
if Git.is_repo(root_dir):
return Git(root_dir)
def SCM(root_dir, no_scm=False, project=None):
if Git.is_repo(root_dir) or Git.is_submodule(root_dir):
return Git(root_dir, project=project)
return Base(root_dir)
return Base(root_dir, project=project)
-17
View File
@@ -1,17 +0,0 @@
import signal
import threading
from dvc.logger import Logger
class SignalHandler(object):
def __enter__(self):
if isinstance(threading.current_thread(), threading._MainThread):
self.old_handler = signal.signal(signal.SIGINT, self.handler)
def handler(self, sig, frame):
Logger.debug('Ignoring SIGINT during critical parts of code...')
def __exit__(self, type, value, traceback):
if isinstance(threading.current_thread(), threading._MainThread):
signal.signal(signal.SIGINT, self.old_handler)
+79 -30
View File
@@ -1,6 +1,5 @@
import os
import yaml
import itertools
import posixpath
import subprocess
from schema import Schema, SchemaError, Optional, Or, And
@@ -8,7 +7,6 @@ from schema import Schema, SchemaError, Optional, Or, And
import dvc.dependency as dependency
import dvc.output as output
from dvc.exceptions import DvcException
from dvc.prompt import prompt
from dvc.logger import Logger
from dvc.utils import dict_md5, fix_env
@@ -20,8 +18,9 @@ class StageCmdFailedError(DvcException):
class StageFileFormatError(DvcException):
def __init__(self):
super(StageFileFormatError, self).__init__('Stage file format error')
def __init__(self, fname, e):
msg = "Stage file '{}' format error: {}".format(fname, str(e))
super(StageFileFormatError, self).__init__(msg)
class StageFileDoesNotExistError(DvcException):
@@ -37,10 +36,21 @@ class StageFileIsNotDvcFileError(DvcException):
class StageFileBadNameError(DvcException):
def __init__(self, fname):
msg = "Bad stage filename '{}'. Stage files should be named " \
"'Dvcfile' or have a '.dvc' suffix(e.g. '{}.dvc')."
super(StageFileBadNameError, self).__init__(msg.format(fname, fname))
def __init__(self, msg):
super(StageFileBadNameError, self).__init__(msg)
class MissingDep(DvcException):
def __init__(self, deps):
assert len(deps) > 0
if len(deps) > 1:
dep = 'dependencies'
else:
dep = 'dependency'
msg = u'missing {}: {}'.format(dep, ', '.join(map(str, deps)))
super(MissingDep, self).__init__(msg)
class MissingDataSource(DvcException):
@@ -124,8 +134,6 @@ class Stage(object):
if self.md5 and md5 and self.md5 == md5:
return False
msg = "Dvc file '{}' md5 changed(expected '{}', actual '{}')"
self.project.logger.debug(msg.format(self.relpath, self.md5, md5))
return True
@property
@@ -138,30 +146,51 @@ class Stage(object):
len(self.deps) == 1 and \
len(self.outs) == 1
def changed(self):
def changed(self, print_info=False):
ret = False
if print_info:
log = self.project.logger.info
else:
log = self.project.logger.debug
if self.is_callback:
msg = "Dvc file '{}' is a 'callback' stage (has a command and " \
"no dependencies) and thus always considered as changed."
self.project.logger.warn(msg.format(self.relpath))
ret = True
if self.locked:
entries = self.outs
else:
entries = itertools.chain(self.outs, self.deps)
for entry in entries:
if entry.changed():
if not self.locked:
for dep in self.deps:
if not dep.changed():
continue
if print_info:
msg = "Dependency '{}' of '{}' changed."
log(msg.format(dep, self.relpath))
ret = True
for out in self.outs:
if not out.changed():
continue
if print_info:
msg = "Output '{}' of '{}' changed."
log(msg.format(out, self.relpath))
ret = True
if self.changed_md5():
if print_info:
msg = "Dvc file '{}' changed."
log(msg.format(self.relpath))
ret = True
if ret:
msg = u'Dvc file \'{}\' changed'.format(self.relpath)
msg = "Stage '{}' changed.".format(self.relpath)
color = 'yellow'
else:
msg = u'Dvc file \'{}\' didn\'t change'.format(self.relpath)
msg = "Stage '{}' didn't change.".format(self.relpath)
color = 'green'
self.project.logger.debug(msg)
log(Logger.colorize(msg, color))
return ret
@@ -174,7 +203,7 @@ class Stage(object):
os.unlink(self.path)
def reproduce(self, force=False, dry=False, interactive=False):
if not self.changed() and not force:
if not self.changed(print_info=True) and not force:
return None
if (self.cmd or self.is_import) and not self.locked and not dry:
@@ -183,7 +212,8 @@ class Stage(object):
msg = "Going to reproduce '{}'. Are you sure you want to continue?"
msg = msg.format(self.relpath)
if interactive and not prompt(msg):
if interactive \
and not self.project.prompt.prompt(msg):
raise DvcException('Reproduction aborted by the user')
self.project.logger.info(u'Reproducing \'{}\''.format(self.relpath))
@@ -196,17 +226,15 @@ class Stage(object):
return self
@staticmethod
def validate(d):
def validate(d, fname=None):
try:
Schema(Stage.SCHEMA).validate(d)
except SchemaError as exc:
Logger.debug(str(exc))
raise StageFileFormatError()
raise StageFileFormatError(fname, exc)
@staticmethod
def loadd(project, d, path):
Stage.validate(d)
Stage.validate(d, fname=os.path.relpath(path))
path = os.path.abspath(path)
cwd = os.path.dirname(path)
cmd = d.get(Stage.PARAM_CMD, None)
@@ -270,6 +298,11 @@ class Stage(object):
use_cache=False, metric=True)
stage.deps = dependency.loads_from(stage, deps)
if fname is not None and os.path.basename(fname) != fname:
msg = "Stage file name '{}' should not contain subdirectories. " \
"Use '-c|--cwd' to change location of the stage file."
raise StageFileBadNameError(msg.format(fname))
fname, cwd = Stage._stage_fname_cwd(fname, cwd, stage.outs, add=add)
cwd = os.path.abspath(cwd)
@@ -279,7 +312,8 @@ class Stage(object):
relpath = os.path.relpath(path)
msg = "'{}' already exists. " \
"Do you wish to run the command and overwrite it?"
if not overwrite and not prompt(msg.format(relpath), False):
if not overwrite \
and not project.prompt.prompt(msg.format(relpath), False):
raise DvcException("'{}' already exists".format(relpath))
stage.cwd = cwd
@@ -290,7 +324,10 @@ class Stage(object):
@staticmethod
def _check_dvc_filename(fname):
if not Stage.is_stage_filename(fname):
raise StageFileBadNameError(os.path.relpath(fname))
msg = "Bad stage filename '{}'. Stage files should be named " \
"'Dvcfile' or have a '.dvc' suffix(e.g. '{}.dvc')."
raise StageFileBadNameError(msg.format(os.path.relpath(fname),
os.path.basename(fname)))
@staticmethod
def _check_dvc_file(fname):
@@ -346,6 +383,8 @@ class Stage(object):
with open(fname, 'w') as fd:
yaml.safe_dump(self.dumpd(), fd, default_flow_style=False)
self.project._files_to_git_add.append(os.path.relpath(fname))
def save(self):
for dep in self.deps:
dep.save()
@@ -353,6 +392,15 @@ class Stage(object):
for out in self.outs:
out.save()
def _check_missing_deps(self):
missing = []
for dep in self.deps:
if not dep.exists:
missing.append(dep)
if len(missing) > 0:
raise MissingDep(missing)
def run(self, dry=False):
if self.locked:
msg = u'Verifying outputs in locked stage \'{}\''
@@ -376,6 +424,7 @@ class Stage(object):
self.project.logger.info(msg)
if not dry:
self._check_missing_deps()
p = subprocess.Popen(self.cmd,
cwd=self.cwd,
shell=True,
+230 -141
View File
@@ -1,60 +1,68 @@
import os
import json
import time
import sqlite3
import nanotime
import threading
from dvc.lock import Lock
from dvc.config import Config
from dvc.system import System
from dvc.utils import file_md5, remove
from dvc.exceptions import DvcException
from dvc.signal_handler import SignalHandler
from dvc.logger import Logger
class StateEntry(object):
PARAM_MTIME = 'mtime'
PARAM_MD5 = 'md5'
def __init__(self, md5, mtime):
self.mtime = mtime
self.md5 = md5
def update(self, md5, mtime):
self.mtime = mtime
self.md5 = md5
@staticmethod
def loadd(d):
mtime = d[StateEntry.PARAM_MTIME]
md5 = d[StateEntry.PARAM_MD5]
return StateEntry(md5, mtime)
def dumpd(self):
return {
self.PARAM_MD5: self.md5,
self.PARAM_MTIME: self.mtime,
}
class StateDuplicateError(DvcException):
pass
class State(object):
VERSION = 1
STATE_FILE = 'state'
STATE_LOCK_FILE = 'state.lock'
STATE_TABLE = 'state'
STATE_TABLE_LAYOUT = "inode UNSIGNED INTEGER PRIMARY KEY, " \
"mtime TEXT NOT NULL, " \
"md5 TEXT NOT NULL, " \
"timestamp TEXT NOT NULL"
def __init__(self, project):
STATE_INFO_TABLE = 'state_info'
STATE_INFO_TABLE_LAYOUT = 'count INTEGER'
STATE_INFO_ROW = 1
LINK_STATE_TABLE = 'link_state'
LINK_STATE_TABLE_LAYOUT = "path TEXT PRIMARY KEY, " \
"inode UNSIGNED INTEGER NOT NULL, " \
"mtime TEXT NOT NULL"
STATE_ROW_LIMIT = 10000000
STATE_ROW_CLEANUP_QUOTA = 50
def __init__(self, project, config):
self.project = project
self.dvc_dir = project.dvc_dir
self._lock = threading.Lock()
if self.dvc_dir:
self.state_file = os.path.join(self.dvc_dir, self.STATE_FILE)
self._lock_file = Lock(self.dvc_dir, self.STATE_LOCK_FILE)
else:
self.root_dir = project.root_dir
self.row_limit = 100
self.row_cleanup_quota = 50
c = config.get(Config.SECTION_STATE, {})
self.row_limit = c.get(Config.SECTION_STATE_ROW_LIMIT,
self.STATE_ROW_LIMIT)
self.row_cleanup_quota = c.get(Config.SECTION_STATE_ROW_CLEANUP_QUOTA,
self.STATE_ROW_CLEANUP_QUOTA)
if not self.dvc_dir:
self.state_file = None
self._lock_file = threading.Lock()
self._db = self.load()
return
self.state_file = os.path.join(self.dvc_dir, self.STATE_FILE)
self.db = None
self.c = None
self.inserts = 0
def __enter__(self):
self.load()
def __exit__(self, type, value, tb):
self.dump()
@staticmethod
def init(project):
@@ -74,137 +82,220 @@ class State(object):
return actual.split('.')[0] != md5.split('.')[0]
def load(self):
if not self.state_file or not os.path.isfile(self.state_file):
return {}
def _execute(self, cmd):
Logger.debug(cmd)
return self.c.execute(cmd)
with open(self.state_file, 'r') as fd:
def _load(self):
from dvc import VERSION
cmd = "PRAGMA user_version;"
self._execute(cmd)
ret = self.c.fetchall()
assert len(ret) == 1
assert len(ret[0]) == 1
assert isinstance(ret[0][0], int)
self.version = ret[0][0]
if self.version > self.VERSION:
msg = "You are using an old version '{}' of dvc that is using " \
"state file version '{}' which is not compatible with the " \
"state file version '{}' that is used in this projet. " \
"Please upgrade right now!"
raise DvcException(msg.format(VERSION,
self.VERSION,
self.version))
elif self.version < self.VERSION:
msg = "State file version '{}' is too old. " \
"Reformatting to the current version '{}'."
self.project.logger.warn(msg.format(self.version, self.VERSION))
cmd = "DROP TABLE IF EXISTS {};"
self._execute(cmd.format(self.STATE_TABLE))
self._execute(cmd.format(self.STATE_INFO_TABLE))
self._execute(cmd.format(self.LINK_STATE_TABLE))
# Check that the state file is indeed a database
cmd = "CREATE TABLE IF NOT EXISTS {} ({})"
self._execute(cmd.format(self.STATE_TABLE,
self.STATE_TABLE_LAYOUT))
self._execute(cmd.format(self.STATE_INFO_TABLE,
self.STATE_INFO_TABLE_LAYOUT))
self._execute(cmd.format(self.LINK_STATE_TABLE,
self.LINK_STATE_TABLE_LAYOUT))
cmd = "INSERT OR IGNORE INTO {} (count) SELECT 0 " \
"WHERE NOT EXISTS (SELECT * FROM {})"
self._execute(cmd.format(self.STATE_INFO_TABLE,
self.STATE_INFO_TABLE))
cmd = "PRAGMA user_version = {};"
self._execute(cmd.format(self.VERSION))
def load(self):
retries = 1
while True:
assert self.db is None
assert self.c is None
assert self.inserts == 0
self.db = sqlite3.connect(self.state_file)
self.c = self.db.cursor()
# Try loading once to check that the file is indeed a database
# and reformat it if it is not.
try:
return json.load(fd)
except ValueError as exc:
Logger.error('Failed to load \'{}\''.format(self.state_file),
exc)
return {}
self._load()
return
except sqlite3.DatabaseError:
self.c.close()
self.db.close()
self.db = None
self.c = None
self.inserts = 0
if retries > 0:
os.unlink(self.state_file)
retries -= 1
else:
raise
def dump(self):
if not self.state_file:
return
assert self.db is not None
with SignalHandler():
with open(self.state_file, 'w+') as fd:
json.dump(self._db, fd)
cmd = "SELECT count from {} WHERE rowid={}"
self._execute(cmd.format(self.STATE_INFO_TABLE,
self.STATE_INFO_ROW))
ret = self.c.fetchall()
assert len(ret) == 1
assert len(ret[0]) == 1
count = ret[0][0] + self.inserts
if count > self.row_limit:
msg = "Cleaning up state. This might take a while."
self.project.logger.warn(msg)
delete = (count - self.row_limit)
delete += int(self.row_limit * (self.row_cleanup_quota/100.))
cmd = "DELETE FROM {} WHERE timestamp IN (" \
"SELECT timestamp FROM {} ORDER BY timestamp ASC LIMIT {});"
self._execute(cmd.format(self.STATE_TABLE,
self.STATE_TABLE,
delete))
self._execute("VACUUM")
cmd = "SELECT COUNT(*) FROM {}"
self._execute(cmd.format(self.STATE_TABLE))
ret = self.c.fetchall()
assert len(ret) == 1
assert len(ret[0]) == 1
count = ret[0][0]
cmd = "UPDATE {} SET count = {} WHERE rowid = {}"
self._execute(cmd.format(self.STATE_INFO_TABLE,
count,
self.STATE_INFO_ROW))
self.db.commit()
self.c.close()
self.db.close()
self.db = None
self.c = None
self.inserts = 0
@staticmethod
def mtime(path):
return str(int(nanotime.timestamp(os.path.getmtime(path))))
mtime = os.path.getmtime(path)
if os.path.isdir(path):
for root, dirs, files in os.walk(path):
for entry in list(dirs) + list(files):
p = os.path.join(root, entry)
m = os.path.getmtime(p)
if m > mtime:
mtime = m
return str(int(nanotime.timestamp(mtime)))
@staticmethod
def inode(path):
return str(System.inode(path))
return System.inode(path)
def _do_update(self, path, dump=True):
def _do_update(self, path):
if not os.path.exists(path):
return (None, None)
mtime = self.mtime(path)
inode = self.inode(path)
md5 = self._get(inode, mtime)
if md5:
return (md5, None)
cmd = 'SELECT * from {} WHERE inode={}'.format(self.STATE_TABLE,
inode)
md5, info = self._collect(path)
state = StateEntry(md5, mtime)
d = state.dumpd()
with self._lock:
with self._lock_file:
self._db[inode] = d
if dump:
self.dump()
self._execute(cmd)
ret = self.c.fetchall()
if len(ret) == 0:
md5, info = self._collect(path)
cmd = 'INSERT INTO {}(inode, mtime, md5, timestamp) ' \
'VALUES ({}, "{}", "{}", "{}")'
self._execute(cmd.format(self.STATE_TABLE,
inode,
mtime,
md5,
int(nanotime.timestamp(time.time()))))
self.inserts += 1
else:
assert len(ret) == 1
assert len(ret[0]) == 4
i, m, md5, timestamp = ret[0]
assert i == inode
if mtime != m:
md5, info = self._collect(path)
cmd = 'UPDATE {} SET ' \
'mtime = "{}", md5 = "{}", timestamp = "{}" ' \
'WHERE inode = {}'
self._execute(cmd.format(self.STATE_TABLE,
mtime,
md5,
int(nanotime.timestamp(time.time())),
inode))
else:
info = None
cmd = 'UPDATE {} SET timestamp = "{}" WHERE inode = {}'
self._execute(cmd.format(self.STATE_TABLE,
int(nanotime.timestamp(time.time())),
inode))
return (md5, info)
def update(self, path, dump=True):
return self._do_update(path, dump=dump)[0]
def update(self, path):
return self._do_update(path)[0]
def update_info(self, path, dump=True):
md5, info = self._do_update(path, dump=dump)
def update_info(self, path):
md5, info = self._do_update(path)
if not info:
info = self.project.cache.local.load_dir_cache(md5)
return (md5, info)
def _get(self, inode, mtime):
with self._lock:
with self._lock_file:
d = self._db.get(inode, None)
if not d:
return None
state = StateEntry.loadd(d)
if mtime == state.mtime:
return state.md5
return None
def get(self, path):
mtime = self.mtime(path)
inode = self.inode(path)
return self._get(inode, mtime)
class LinkStateEntry(object):
PARAM_MTIME = 'mtime'
PARAM_INODE = 'inode'
def __init__(self, inode, mtime):
self.mtime = mtime
self.inode = inode
@staticmethod
def loadd(d):
mtime = d[LinkStateEntry.PARAM_MTIME]
inode = d[LinkStateEntry.PARAM_INODE]
return LinkStateEntry(inode, mtime)
def dumpd(self):
return {
self.PARAM_INODE: self.inode,
self.PARAM_MTIME: self.mtime,
}
class LinkState(State):
STATE_FILE = 'link.state'
STATE_LOCK_FILE = STATE_FILE + '.lock'
def __init__(self, project):
super(LinkState, self).__init__(project)
self.root_dir = project.root_dir
def update(self, path, dump=True):
def update_link(self, path):
if not os.path.exists(path):
return
mtime = self.mtime(path)
inode = self.inode(path)
relpath = os.path.relpath(path, self.root_dir)
with self._lock:
state = LinkStateEntry(inode, mtime)
d = state.dumpd()
self._db[os.path.relpath(path, self.root_dir)] = d
if dump:
with self._lock_file:
self.dump()
cmd = 'REPLACE INTO {}(path, inode, mtime) ' \
'VALUES ("{}", {}, "{}")'.format(self.LINK_STATE_TABLE,
relpath,
inode,
mtime)
self._execute(cmd)
def _do_remove_unused(self, used):
items = self._db.copy().items()
for p, s in items:
def remove_unused_links(self, used):
unused = []
self._execute('SELECT * FROM {}'.format(self.LINK_STATE_TABLE))
for row in self.c:
p, i, m = row
path = os.path.join(self.root_dir, p)
state = LinkStateEntry.loadd(s)
if path in used:
continue
@@ -215,13 +306,11 @@ class LinkState(State):
inode = self.inode(path)
mtime = self.mtime(path)
if inode == state.inode and mtime == state.mtime:
if i == inode and m == mtime:
Logger.debug('Removing \'{}\' as unused link.'.format(path))
remove(path)
del self._db[p]
unused.append(p)
def remove_unused(self, used):
with self._lock:
with self._lock_file:
self._do_remove_unused(used)
self.dump()
for p in unused:
cmd = 'DELETE FROM {} WHERE path = "{}"'
self._execute(cmd.format(self.LINK_STATE_TABLE, p))
+135 -24
View File
@@ -1,9 +1,4 @@
import os
import ctypes
import reflink
if os.name == 'nt':
import ntfsutils.hardlink as winlink
class System(object):
@@ -13,13 +8,24 @@ class System(object):
@staticmethod
def hardlink(source, link_name):
import ctypes
if System.is_unix():
return os.link(source, link_name)
os.link(source, link_name)
return
return winlink.create(source, link_name)
CreateHardLink = ctypes.windll.kernel32.CreateHardLinkW
CreateHardLink.argtypes = [ctypes.c_wchar_p,
ctypes.c_wchar_p,
ctypes.c_void_p]
CreateHardLink.restype = ctypes.wintypes.BOOL
res = CreateHardLink(link_name, source, None)
if res == 0:
raise ctypes.WinError()
@staticmethod
def symlink(source, link_name):
import ctypes
if System.is_unix():
return os.symlink(source, link_name)
@@ -35,33 +41,138 @@ class System(object):
raise ctypes.WinError()
@staticmethod
def _reflink_darwin(src, dst):
import ctypes
clib = ctypes.CDLL('libc.dylib')
if not hasattr(clib, 'clonefile'):
return -1
clonefile = clib.clonefile
clonefile.argtypes = [ctypes.c_char_p,
ctypes.c_char_p,
ctypes.c_int]
clonefile.restype = ctypes.c_int
return clonefile(ctypes.c_char_p(src.encode('utf-8')),
ctypes.c_char_p(dst.encode('utf-8')),
ctypes.c_int(0))
@staticmethod
def _reflink_windows(src, dst):
return -1
@staticmethod
def _reflink_linux(src, dst):
import os
import fcntl
FICLONE = 0x40049409
s = open(src, 'r')
d = open(dst, 'w+')
try:
ret = fcntl.ioctl(d.fileno(), FICLONE, s.fileno())
except IOError:
s.close()
d.close()
os.unlink(dst)
raise
s.close()
d.close()
if ret != 0:
os.unlink(dst)
return ret
@staticmethod
def reflink(source, link_name):
return reflink.reflink(source, link_name)
import platform
from dvc.exceptions import DvcException
system = platform.system()
try:
if system == 'Windows':
ret = System._reflink_windows(source, link_name)
elif system == 'Darwin':
ret = System._reflink_darwin(source, link_name)
elif system == 'Linux':
ret = System._reflink_linux(source, link_name)
else:
ret = -1
except IOError:
ret = -1
if ret != 0:
raise DvcException('Reflink is not supported')
# FIXME
# Temporary fix while waiting for the PR to be merged and released:
# https://github.com/sunshowers/ntfs/pull/11
@staticmethod
def getdirinfo(path):
from ntfsutils.fs import FILE_FLAG_BACKUP_SEMANTICS, FILE_SHARE_READ
from ntfsutils.fs import OPEN_EXISTING, BY_HANDLE_FILE_INFORMATION
from ntfsutils.fs import CreateFile, WinError
from ntfsutils.fs import GetFileInformationByHandle, CloseHandle
import ctypes
from ctypes import c_void_p, c_wchar_p, Structure, WinError
from ctypes.wintypes import DWORD, HANDLE, POINTER, BOOL
FILE_FLAG_BACKUP_SEMANTICS = 0x02000000
FILE_SHARE_READ = 0x00000001
OPEN_EXISTING = 3
class FILETIME(Structure):
_fields_ = [("dwLowDateTime", DWORD),
("dwHighDateTime", DWORD)]
class BY_HANDLE_FILE_INFORMATION(Structure):
_fields_ = [("dwFileAttributes", DWORD),
("ftCreationTime", FILETIME),
("ftLastAccessTime", FILETIME),
("ftLastWriteTime", FILETIME),
("dwVolumeSerialNumber", DWORD),
("nFileSizeHigh", DWORD),
("nFileSizeLow", DWORD),
("nNumberOfLinks", DWORD),
("nFileIndexHigh", DWORD),
("nFileIndexLow", DWORD)]
flags = FILE_FLAG_BACKUP_SEMANTICS
hfile = CreateFile(path,
0,
FILE_SHARE_READ,
None,
OPEN_EXISTING,
flags,
None)
func = ctypes.windll.kernel32.CreateFileW
func.argtypes = [c_wchar_p,
DWORD,
DWORD,
c_void_p,
DWORD,
DWORD,
HANDLE]
func.restype = HANDLE
hfile = func(path,
0,
FILE_SHARE_READ,
None,
OPEN_EXISTING,
flags,
None)
if hfile is None:
raise WinError()
func = ctypes.windll.kernel32.GetFileInformationByHandle
func.argtypes = [HANDLE, POINTER(BY_HANDLE_FILE_INFORMATION)]
func.restype = BOOL
info = BY_HANDLE_FILE_INFORMATION()
rv = GetFileInformationByHandle(hfile, info)
CloseHandle(hfile)
rv = func(hfile, info)
func = ctypes.windll.kernel32.CloseHandle
func.argtypes = [HANDLE]
func.restype = BOOL
func(hfile)
if rv == 0:
raise WinError()
return info
@staticmethod
+2 -2
View File
@@ -6,11 +6,11 @@ from dvc import VERSION_BASE
from dvc.logger import Logger
class Updater(object):
class Updater(object): # pragma: no cover
URL = 'https://4ki8820rsf.execute-api.us-east-2.amazonaws.com/' \
'prod/latest-version'
UPDATER_FILE = 'updater'
TIMEOUT = 7 * 24 * 60 * 60 # every week
TIMEOUT = 24 * 60 * 60 # every day
TIMEOUT_GET = 10
def __init__(self, dvc_dir):
+4 -5
View File
@@ -4,23 +4,22 @@ ply>=3.9 # See https://github.com/pyinstaller/pyinstaller/issues/1945
configparser>=3.5.0
zc.lockfile>=1.2.1
future>=0.16.0
google-cloud==0.32.0
google-cloud-storage==1.12.0
PyInstaller==3.3.1
colorama>=0.3.9
configobj>=5.0.6
networkx==2.1
networkx>=2.1
pyyaml>=3.12
gitpython>=2.1.8
ntfsutils>=0.1.4
setuptools>=34.0.0
nanotime>=0.5.2
pyasn1>=0.4.1
schema>=0.6.7
paramiko>=2.4.1
jsonpath-rw==1.4.0
reflink==0.2.0
requests>=2.18.4
wheel>=0.31.1
futures>=3.2.0; python_version == "2.7"
grandalf==0.6
asciicanvas==0.0.3
pydot>=1.2.4
asciimatics>=1.10.0
+1 -1
View File
@@ -14,7 +14,7 @@ function retry {
# NOTE: it is not uncommon for pip to hang on travis for what seems to be
# networking issues. Thus, let's retry a few times to see if it will eventially
# work or not.
retry pip install --upgrade pip
retry pip install --upgrade pip setuptools wheel
retry pip install -r requirements.txt
retry pip install -r test-requirements.txt
+2 -1
View File
@@ -6,7 +6,8 @@ set -e
python -mtests
if [[ "$TRAVIS_PULL_REQUEST" == "false" && \
"$TRAVIS_SECURE_ENV_VARS" == "true" ]]; then
"$TRAVIS_SECURE_ENV_VARS" == "true" && \
"$(python -c 'import sys; print(sys.version_info[0])')" == '2' ]]; then
codecov
fi
+5 -7
View File
@@ -12,24 +12,22 @@ install_requires = [
"future>=0.16.0",
"colorama>=0.3.9",
"configobj>=5.0.6",
"networkx==2.1",
"networkx>=2.1",
"pyyaml>=3.12",
"gitpython>=2.1.8",
"ntfsutils>=0.1.4",
"setuptools>=34.0.0",
"nanotime>=0.5.2",
"pyasn1>=0.4.1",
"schema>=0.6.7",
"jsonpath-rw==1.4.0",
"reflink==0.2.0",
"requests>=2.18.4",
"grandalf==0.6",
"asciicanvas==0.0.3",
"asciimatics>=1.10.0",
]
# Extra dependencies for remote integrations
gs = [
"google-cloud==0.32.0",
"google-cloud-storage==1.12.0",
]
s3 = [
"boto3==1.7.4",
@@ -67,11 +65,11 @@ setup(
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 3',
],
packages=find_packages(exclude=['bin', 'tests', 'functests']),
packages=find_packages(exclude=['tests']),
include_package_data=True,
url='http://dataversioncontrol.com',
entry_points={
'console_scripts': ['dvc = dvc.main:main']
},
zip_safe=False
zip_safe=False,
)
+4 -2
View File
@@ -1,9 +1,11 @@
pytest>=3.3.1
nose>=1.3.7
mock>=2.0.0
coverage
codecov
coverage>=4.5.1
codecov>=2.0.15
xmltodict>=0.11.0
awscli>=1.14.18
google-compute-engine
pywin32; sys_platform == 'win32'
Pygments
collective.checkdocs
+2 -1
View File
@@ -7,4 +7,5 @@ os.chdir(REPO_ROOT)
os.putenv('PATH', '{}:{}'.format(os.path.join(REPO_ROOT, 'bin'), os.getenv('PATH')))
os.putenv('DVC_HOME', REPO_ROOT)
check_call('nosetests -v --processes=-1 --process-timeout=200 --cover-inclusive --cover-erase --cover-package=dvc --with-coverage', shell=True)
cmd = 'nosetests -v --processes=-1 --process-timeout=200 --cover-inclusive --cover-erase --cover-package=dvc --with-coverage'
check_call(cmd, shell=True)
+9
View File
@@ -61,6 +61,15 @@ class TestGit(TestDir):
self.git.index.commit('add code')
class TestGitSubmodule(TestGit):
def setUp(self):
super(TestGitSubmodule, self).setUp()
subrepo = Repo.init()
subrepo_path = 'subrepo'
self.git.create_submodule(subrepo_path, subrepo_path, subrepo.git_dir)
self._pushd(subrepo_path)
class TestDvc(TestGit):
def setUp(self):
super(TestDvc, self).setUp()
+9
View File
@@ -103,3 +103,12 @@ class TestSharedCacheDir(TestDir):
self.assertEqual(len(os.listdir(os.path.join(cache_dir, subdirs[0]))), 1)
self.assertEqual(len(os.listdir(os.path.join(cache_dir, subdirs[1]))), 1)
self.assertEqual(len(os.listdir(os.path.join(cache_dir, subdirs[2]))), 1)
class TestCacheLinkType(TestDvc):
def test(self):
ret = main(['config', 'cache.type', 'reflink,copy'])
self.assertEqual(ret, 0)
ret = main(['add', self.FOO])
self.assertEqual(ret, 0)
+22 -5
View File
@@ -99,7 +99,8 @@ class TestRemoveFilesWhenCheckout(CheckoutBase):
# add the file into a separate branch
self.dvc.scm.checkout(branch_1, True)
main(['checkout'])
ret = main(['checkout'])
self.assertEqual(ret, 0)
self.commit_data_file(fname)
# Checkout back in master
@@ -108,7 +109,8 @@ class TestRemoveFilesWhenCheckout(CheckoutBase):
# Make sure `dvc checkout` removes the file
# self.dvc.checkout()
main(['checkout'])
ret = main(['checkout'])
self.assertEqual(ret, 0)
self.assertFalse(os.path.exists(fname))
@@ -151,11 +153,13 @@ class TestGitIgnoreWhenCheckout(CheckoutBase):
self.commit_data_file(fname_master)
self.dvc.scm.checkout(branch_1, True)
main(['checkout'])
ret = main(['checkout'])
self.assertEqual(ret, 0)
self.commit_data_file(fname_branch)
self.dvc.scm.checkout(branch_master)
main(['checkout'])
ret = main(['checkout'])
self.assertEqual(ret, 0)
ignored = self.read_ignored()
@@ -163,7 +167,8 @@ class TestGitIgnoreWhenCheckout(CheckoutBase):
self.assertIn(fname_master, ignored)
self.dvc.scm.checkout(branch_1)
main(['checkout'])
ret = main(['checkout'])
self.assertEqual(ret, 0)
ignored = self.read_ignored()
self.assertIn(fname_branch, ignored)
@@ -200,3 +205,15 @@ class TestCheckoutEmptyDir(TestDvc):
self.assertTrue(os.path.isdir(dname))
self.assertEqual(len(os.listdir(dname)), 0)
class TestCheckoutNotCachedFile(TestDvc):
def test(self):
cmd = 'python {} {} {}'.format(self.CODE, self.FOO, 'out')
self.dvc.add(self.FOO)
stage = self.dvc.run(cmd=cmd,
deps=[self.FOO, self.CODE],
outs_no_cache=['out'])
self.dvc.checkout()
+46 -44
View File
@@ -1,5 +1,5 @@
from subprocess import CalledProcessError
from subprocess import check_output
from subprocess import check_output, Popen
from unittest import SkipTest
import os
import time
@@ -94,6 +94,13 @@ def _should_test_hdfs():
except (CalledProcessError, IOError):
return False
p = Popen('hadoop fs -ls hdfs://127.0.0.1/',
shell=True,
executable=os.getenv('SHELL'))
p.communicate()
if p.returncode != 0:
return False
return True
@@ -221,55 +228,56 @@ class TestDataCloudBase(TestDvc):
info_dir = stage_dir.outs[0].dumpd()
md5_dir = info_dir['md5']
# Check status
sleep()
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_NEW)], status)
with self.cloud.project.state:
# Check status
sleep()
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_NEW)], status)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertTrue((md5_dir, STATUS_NEW) in status_dir)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertTrue((md5_dir, STATUS_NEW) in status_dir)
# Push and check status
self.cloud.push([info])
self.assertTrue(os.path.exists(cache))
self.assertTrue(os.path.isfile(cache))
# Push and check status
self.cloud.push([info])
self.assertTrue(os.path.exists(cache))
self.assertTrue(os.path.isfile(cache))
self.cloud.push([info_dir])
self.assertTrue(os.path.isfile(cache_dir))
self.cloud.push([info_dir])
self.assertTrue(os.path.isfile(cache_dir))
sleep()
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_OK)], status)
sleep()
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_OK)], status)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertTrue((md5_dir, STATUS_OK) in status_dir)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertTrue((md5_dir, STATUS_OK) in status_dir)
# Remove and check status
sleep()
shutil.rmtree(self.dvc.cache.local.cache_dir)
# Remove and check status
sleep()
shutil.rmtree(self.dvc.cache.local.cache_dir)
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_DELETED)], status)
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_DELETED)], status)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertEqual([(md5_dir, STATUS_DELETED)], status_dir)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertEqual([(md5_dir, STATUS_DELETED)], status_dir)
# Pull and check status
self.cloud.pull([info])
self.assertTrue(os.path.exists(cache))
self.assertTrue(os.path.isfile(cache))
with open(cache, 'r') as fd:
self.assertEqual(fd.read(), self.FOO_CONTENTS)
# Pull and check status
self.cloud.pull([info])
self.assertTrue(os.path.exists(cache))
self.assertTrue(os.path.isfile(cache))
with open(cache, 'r') as fd:
self.assertEqual(fd.read(), self.FOO_CONTENTS)
self.cloud.pull([info_dir])
self.assertTrue(os.path.isfile(cache_dir))
self.cloud.pull([info_dir])
self.assertTrue(os.path.isfile(cache_dir))
sleep()
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_OK)], status)
sleep()
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_OK)], status)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertTrue((md5_dir, STATUS_OK) in status_dir)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertTrue((md5_dir, STATUS_OK) in status_dir)
def test(self):
self._ensure_should_run()
@@ -335,9 +343,6 @@ class TestRemoteSSH(TestDataCloudBase):
def _get_cloud_class(self):
return RemoteSSH
def test(self):
super(TestRemoteSSH, self).test()
class TestRemoteHDFS(TestDataCloudBase):
def _should_test(self):
@@ -349,9 +354,6 @@ class TestRemoteHDFS(TestDataCloudBase):
def _get_cloud_class(self):
return RemoteHDFS
def test(self):
super(TestRemoteHDFS, self).test()
class TestDataCloudCLIBase(TestDvc):
def main(self, args):
+2 -1
View File
@@ -29,7 +29,8 @@ class TestGC(TestDvc):
self._test_gc()
def test_cli(self):
ret = main(['gc'])
ret = main(['gc', '-f'])
self.assertEqual(ret, 0)
self._test_gc()
def _test_gc(self):
+14 -1
View File
@@ -1,9 +1,10 @@
import os
from dvc.main import main
from dvc.exceptions import DvcException
from dvc.exceptions import DvcException, MoveNotDataSourceError
from tests.basic_env import TestDvc
from tests.test_repro import TestRepro
class TestMove(TestDvc):
@@ -39,3 +40,15 @@ class TestCmdMove(TestDvc):
ret = main(['move', 'non-existing-file', 'dst'])
self.assertNotEqual(ret, 0)
class TestMoveNotDataSource(TestRepro):
def test(self):
from dvc.project import Project
self.dvc = Project(self._root_dir)
with self.assertRaises(MoveNotDataSourceError):
self.dvc.move(self.file1, 'dst')
ret = main(['move', self.file1, 'dst'])
self.assertNotEqual(ret, 0)
+96
View File
@@ -1,9 +1,71 @@
from dvc.main import main
from tests.basic_env import TestDvc
from tests.test_repro import TestRepro, TestReproChangedDeepData
import os
class TestPipelineShowSingle(TestDvc):
def setUp(self):
super(TestPipelineShowSingle, self).setUp()
self.stage = 'foo.dvc'
self.dotFile = 'graph.dot'
ret = main(['add', self.FOO])
self.assertEqual(ret, 0)
def test(self):
ret = main(['pipeline', 'show', self.stage])
self.assertEqual(ret, 0)
def test_commands(self):
ret = main(['pipeline', 'show', self.stage, '--commands'])
self.assertEqual(ret, 0)
def test_outs(self):
ret = main(['pipeline', 'show', self.stage, '--outs'])
self.assertEqual(ret, 0)
def test_ascii(self):
ret = main(['pipeline', 'show', '--ascii', self.stage])
self.assertEqual(ret, 0)
def test_dot(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.stage])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_ascii_commands(self):
ret = main(['pipeline', 'show', '--ascii', self.stage, '--commands'])
self.assertEqual(ret, 0)
def test_ascii_outs(self):
ret = main(['pipeline', 'show', '--ascii', self.stage, '--outs'])
self.assertEqual(ret, 0)
def test_dot_commands(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.stage, '--commands'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_dot_outs(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.stage, '--outs'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_not_dvc_file(self):
ret = main(['pipeline', 'show', self.FOO])
self.assertNotEqual(ret, 0)
def test_non_existing(self):
ret = main(['pipeline', 'show', 'non-existing'])
self.assertNotEqual(ret, 0)
class TestPipelineShow(TestRepro):
def setUp(self):
super(TestPipelineShow, self).setUp()
self.dotFile = 'graph.dot'
def test(self):
ret = main(['pipeline', 'show', self.file1_stage])
self.assertEqual(ret, 0)
@@ -20,6 +82,11 @@ class TestPipelineShow(TestRepro):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage])
self.assertEqual(ret, 0)
def test_dot(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_ascii_commands(self):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage, '--commands'])
self.assertEqual(ret, 0)
@@ -28,6 +95,16 @@ class TestPipelineShow(TestRepro):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage, '--outs'])
self.assertEqual(ret, 0)
def test_dot_commands(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage, '--commands'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_dot_outs(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage, '--outs'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_not_dvc_file(self):
ret = main(['pipeline', 'show', self.file1])
self.assertNotEqual(ret, 0)
@@ -38,6 +115,10 @@ class TestPipelineShow(TestRepro):
class TestPipelineShowDeep(TestReproChangedDeepData):
def setUp(self):
super(TestPipelineShowDeep, self).setUp()
self.dotFile = 'graph.dot'
def test(self):
ret = main(['pipeline', 'show', self.file1_stage])
self.assertEqual(ret, 0)
@@ -54,6 +135,11 @@ class TestPipelineShowDeep(TestReproChangedDeepData):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage])
self.assertEqual(ret, 0)
def test_dot(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_ascii_commands(self):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage, '--commands'])
self.assertEqual(ret, 0)
@@ -62,6 +148,16 @@ class TestPipelineShowDeep(TestReproChangedDeepData):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage, '--outs'])
self.assertEqual(ret, 0)
def test_dot_commands(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage, '--commands'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_dot_outs(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage, '--outs'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_not_dvc_file(self):
ret = main(['pipeline', 'show', self.file1])
self.assertNotEqual(ret, 0)
+44
View File
@@ -1,4 +1,8 @@
import os
import configobj
from dvc.main import main
from dvc.project import Project
from dvc.command.remote import CmdRemoteAdd
from dvc.exceptions import UnsupportedRemoteError
from dvc.remote import Remote
@@ -41,3 +45,43 @@ class TestRemote(TestDvc):
def test_unsupported(self):
with self.assertRaises(UnsupportedRemoteError):
Remote(self.dvc, {Config.SECTION_REMOTE_URL: 'unsupported://url'})
class TestRemoteRemoveDefault(TestDvc):
def test(self):
remote = 'mys3'
self.assertEqual(main(['remote', 'add', '--default', remote, 's3://bucket/name']), 0)
self.assertEqual(main(['remote', 'modify', remote, 'profile', 'default']), 0)
self.assertEqual(main(['config', '--local', 'core.remote', remote]), 0)
config = configobj.ConfigObj(os.path.join(self.dvc.dvc_dir, Config.CONFIG))
local_config = configobj.ConfigObj(os.path.join(self.dvc.dvc_dir, Config.CONFIG_LOCAL))
self.assertEqual(config[Config.SECTION_CORE][Config.SECTION_CORE_REMOTE], remote)
self.assertEqual(local_config[Config.SECTION_CORE][Config.SECTION_CORE_REMOTE], remote)
self.assertEqual(main(['remote', 'remove', remote]), 0)
config = configobj.ConfigObj(os.path.join(self.dvc.dvc_dir, Config.CONFIG))
local_config = configobj.ConfigObj(os.path.join(self.dvc.dvc_dir, Config.CONFIG_LOCAL))
section = Config.SECTION_REMOTE_FMT.format(remote)
self.assertTrue(section not in config.keys())
core = config.get(Config.SECTION_CORE, None)
if core is not None:
self.assertTrue(Config.SECTION_CORE_REMOTE not in core.keys())
core = local_config.get(Config.SECTION_CORE, None)
if core is not None:
self.assertTrue(Config.SECTION_CORE_REMOTE not in core.keys())
class TestRemoteRemove(TestDvc):
def test(self):
ret = main(['config', 'core.jobs', '1'])
self.assertEqual(ret, 0)
remote = 'mys3'
ret = main(['remote', 'add', remote, 's3://bucket/name'])
self.assertEqual(ret, 0)
ret = main(['remote', 'remove', remote])
self.assertEqual(ret, 0)
+147
View File
@@ -18,6 +18,7 @@ from dvc.project import Project, ReproductionError
from dvc.utils import file_md5
from dvc.remote.local import RemoteLOCAL
from dvc.stage import Stage, StageFileDoesNotExistError
from dvc.system import System
from tests.basic_env import TestDvc
from tests.test_data_cloud import _should_test_aws, TEST_AWS_REPO_BUCKET
@@ -145,6 +146,48 @@ class TestReproDry(TestReproChangedData):
self.assertFalse(filecmp.cmp(self.file1, self.BAR, shallow=False))
class TestReproUpToDate(TestRepro):
def test(self):
ret = main(['repro', self.file1_stage])
self.assertEqual(ret, 0)
class TestReproDryNoExec(TestDvc):
def test(self):
deps = []
for d in range(3):
idir = 'idir{}'.format(d)
odir = 'odir{}'.format(d)
deps.append('-d')
deps.append(odir)
os.mkdir(idir)
f = os.path.join(idir, 'file')
with open(f, 'w+') as fobj:
fobj.write(str(d))
if os.name == 'nt':
cp = 'copy'
else:
cp = 'cp'
ret = main(['run',
'--no-exec',
'-d', idir,
'-o', odir,
"python -c 'import shutil; "
"shutil.copytree(\"{}\", \"{}\")'".format(idir, odir)])
self.assertEqual(ret, 0)
ret = main(['run', '--no-exec', '-f', 'Dvcfile'] + deps)
self.assertEqual(ret, 0)
ret = main(['repro', '--dry'])
self.assertEqual(ret, 0)
class TestReproChangedDeepData(TestReproChangedData):
def setUp(self):
super(TestReproChangedDeepData, self).setUp()
@@ -168,6 +211,61 @@ class TestReproChangedDeepData(TestReproChangedData):
self.assertEqual(len(stages), 3)
class TestReproPipeline(TestReproChangedDeepData):
def test(self):
stages = self.dvc.reproduce(self.file1_stage,
force=True,
pipeline=True)
self.assertEqual(len(stages), 3)
def test_cli(self):
ret = main(['repro', '--pipeline', '-f', self.file1_stage])
self.assertEqual(ret, 0)
class TestReproPipelines(TestDvc):
def setUp(self):
super(TestReproPipelines, self).setUp()
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
self.foo_stage = stages[0]
self.assertTrue(self.foo_stage is not None)
stages = self.dvc.add(self.BAR)
self.assertEqual(len(stages), 1)
self.bar_stage = stages[0]
self.assertTrue(self.bar_stage is not None)
self.file1 = 'file1'
self.file1_stage = self.file1 + '.dvc'
self.dvc.run(fname=self.file1_stage,
outs=[self.file1],
deps=[self.FOO, self.CODE],
cmd='python {} {} {}'.format(self.CODE, self.FOO, self.file1))
self.file2 = 'file2'
self.file2_stage = self.file2 + '.dvc'
self.dvc.run(fname=self.file2_stage,
outs=[self.file2],
deps=[self.BAR, self.CODE],
cmd='python {} {} {}'.format(self.CODE, self.BAR, self.file2))
def test(self):
stages = self.dvc.reproduce(all_pipelines=True, force=True)
self.assertEqual(len(stages), 4)
names = [stage.relpath for stage in stages]
self.assertTrue(self.foo_stage.relpath in names)
self.assertTrue(self.bar_stage.relpath in names)
self.assertTrue(self.file1_stage in names)
self.assertTrue(self.file2_stage in names)
def test_cli(self):
ret = main(['repro', '-f', '-P'])
self.assertEqual(ret, 0)
class TestReproLocked(TestReproChangedData):
def test(self):
file2 = 'file2'
@@ -255,6 +353,55 @@ class TestReproChangedDir(TestDvc):
self.assertEqual(len(stages), 1)
class TestReproChangedDirData(TestDvc):
def test(self):
dir_name = 'dir'
dir_code = 'dir_code.py'
sleep()
with open(dir_code, 'w+') as fd:
fd.write("import os; import sys; import shutil; shutil.copytree(sys.argv[1], sys.argv[2])")
sleep()
stage = self.dvc.run(outs=[dir_name],
deps=[self.DATA_DIR, dir_code],
cmd="python {} {} {}".format(dir_code,
self.DATA_DIR,
dir_name))
self.assertTrue(stage is not None)
stages = self.dvc.reproduce(stage.path)
self.assertEqual(len(stages), 0)
sleep()
with open(self.DATA_SUB, 'a') as fd:
fd.write('add')
sleep()
stages = self.dvc.reproduce(stage.path)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
sleep()
# Check that dvc indeed registers changed output dir
shutil.move(self.BAR, dir_name)
sleep()
stages = self.dvc.reproduce(stage.path)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
# Check that dvc registers mtime change for the directory.
System.hardlink(self.DATA_SUB, self.DATA_SUB + '.lnk')
sleep()
stages = self.dvc.reproduce(stage.path)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
class TestReproMissingMd5InStageFile(TestRepro):
def test(self):
with open(self.file1_stage, 'r') as fd:
+22 -2
View File
@@ -3,7 +3,7 @@ import filecmp
from dvc.main import main
from dvc.utils import file_md5
from dvc.stage import Stage, StageFileBadNameError
from dvc.stage import Stage, StageFileBadNameError, MissingDep
from dvc.command.run import CmdRun
from dvc.exceptions import OutputDuplicationError
@@ -16,7 +16,7 @@ class TestRun(TestDvc):
deps = [self.FOO, self.CODE]
outs = [os.path.join(self.dvc.root_dir, 'out')]
outs_no_cache = []
fname = os.path.join(self.dvc.root_dir, 'out.dvc')
fname = 'out.dvc'
cwd = os.curdir
self.dvc.add(self.FOO)
@@ -55,6 +55,17 @@ class TestRunEmpty(TestDvc):
cwd=os.curdir)
class TestRunMissingDep(TestDvc):
def test(self):
with self.assertRaises(MissingDep):
self.dvc.run(cmd='',
deps=['non-existing-dep'],
outs=[],
outs_no_cache=[],
fname='empty.dvc',
cwd=os.curdir)
class TestRunBadStageFilename(TestDvc):
def test(self):
with self.assertRaises(StageFileBadNameError):
@@ -65,6 +76,15 @@ class TestRunBadStageFilename(TestDvc):
fname='empty',
cwd=os.curdir)
with self.assertRaises(StageFileBadNameError):
self.dvc.run(cmd='',
deps=[],
outs=[],
outs_no_cache=[],
fname=os.path.join(self.DATA_DIR, 'empty.dvc'),
cwd=os.curdir)
class TestRunNoExec(TestDvc):
def test(self):
+19 -4
View File
@@ -4,7 +4,7 @@ from git import Repo
from dvc.scm import SCM, Base, Git
from tests.basic_env import TestDir, TestGit
from tests.basic_env import TestDir, TestGit, TestGitSubmodule
class TestSCM(TestDir):
@@ -27,6 +27,20 @@ class TestSCMGit(TestGit):
self.assertTrue('foo' in self.git.git.ls_files())
class TestSCMGitSubmodule(TestGitSubmodule):
def test_git_submodule(self):
self.assertIsInstance(SCM(os.curdir), Git)
def test_is_submodule(self):
self.assertTrue(Git.is_submodule(os.curdir))
def test_commit_in_submodule(self):
G = Git(self._root_dir)
G.add(['foo'])
G.commit('add')
self.assertTrue('foo' in self.git.git.ls_files())
class TestIgnore(TestGit):
def _count_gitignore(self):
with open(Git.GITIGNORE, 'r') as fd:
@@ -35,13 +49,14 @@ class TestIgnore(TestGit):
def test(self):
git = Git(self._root_dir)
foo = os.path.join(self._root_dir, self.FOO)
git.ignore('foo')
git.ignore(foo)
self.assertTrue(os.path.isfile(Git.GITIGNORE))
self.assertEqual(self._count_gitignore(), 1)
git.ignore('foo')
git.ignore(foo)
self.assertEqual(self._count_gitignore(), 1)
git.ignore_remove('foo')
git.ignore_remove(foo)
self.assertEqual(self._count_gitignore(), 0)
+19 -24
View File
@@ -3,7 +3,7 @@ import stat
import time
from dvc.system import System
from dvc.state import State, StateEntry
from dvc.state import State
from dvc.utils import file_md5
from tests.basic_env import TestDvc
@@ -16,31 +16,26 @@ class TestState(TestDvc):
mtime = os.path.getmtime(path)
inode = System.inode(path)
state = State(self.dvc)
state = State(self.dvc, self.dvc.config._config)
entry_md5 = state.update(path)
self.assertEqual(entry_md5, md5)
with state:
entry_md5 = state.update(path)
self.assertEqual(entry_md5, md5)
# Sleep some time to simulate realistic behavior.
# Some filesystems have a bad date resolution for
# mtime(i.e. 1sec for HFS) that cause problems with
# our 'state' system not being able to distinguish
# files that were modified within that delta.
time.sleep(1)
# Sleep some time to simulate realistic behavior.
# Some filesystems have a bad date resolution for
# mtime(i.e. 1sec for HFS) that cause problems with
# our 'state' system not being able to distinguish
# files that were modified within that delta.
time.sleep(1)
os.unlink(path)
with open(path, 'w+') as fd:
fd.write('1')
os.unlink(path)
with open(path, 'w+') as fd:
fd.write('1')
md5 = file_md5(path)[0]
mtime = os.path.getmtime(path)
inode = System.inode(path)
md5 = file_md5(path)[0]
mtime = os.path.getmtime(path)
inode = System.inode(path)
entry_md5 = state.update(path)
self.assertEqual(entry_md5, md5)
# Reload state db to make sure that it stays the same
old_db = state._db
state = State(self.dvc)
new_db = state._db
self.assertEqual(old_db, new_db)
entry_md5 = state.update(path)
self.assertEqual(entry_md5, md5)