Compare commits

..
510 Commits
Author SHA1 Message Date
Ruslan Kuprieiev c6811ae005 dvc: bump to 0.19.10 2018-10-12 04:22:26 +03:00
Ruslan Kuprieiev 6ad674ca88 Merge pull request #1213 from efiop/master
state: use UNSIGNED INTEGER for inodes
2018-10-12 04:22:03 +03:00
Ruslan Kuprieiev 45afd36d76 state: use UNSIGNED INTEGER for inodes
Fixes #1207

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-12 03:51:29 +03:00
Ruslan Kuprieiev 0aac01ab88 state: add debug msg
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-12 03:51:29 +03:00
Ruslan Kuprieiev ce3bcc4af9 Merge pull request #1211 from efiop/master
dagascii: import asciimatics only when needed
2018-10-12 00:11:06 +03:00
Ruslan Kuprieiev 06064061d6 dagascii: import asciimatics only when needed
Related to #1210

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-11 23:53:56 +03:00
Ruslan Kuprieiev cae6e712c6 dvc: bump to 0.19.9 2018-10-11 16:11:17 +03:00
Ruslan Kuprieiev 4b525fe1fc system: fix typo in _reflink_darwin 2018-10-11 16:05:27 +03:00
Ruslan Kuprieiev 13a2e4692d Merge pull request #1201 from efiop/master
repro: show which dependencies have changed
2018-10-11 00:10:39 +03:00
Ruslan Kuprieiev 1de87bc91c repro: show which dependencies have changed
Fixes #1164

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-10 23:40:26 +03:00
Ruslan Kuprieiev 82b10a8256 Merge pull request #1200 from efiop/master
import: add a note about supported schemes
2018-10-10 02:59:09 +03:00
Ruslan Kuprieiev a736082595 import: add a note about supported schemes
Fixes #1199

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-10 02:47:20 +03:00
Ruslan Kuprieiev 4888ab82a7 dvc: bump to 0.19.8 2018-10-09 21:01:17 +03:00
Ruslan Kuprieiev bc93cc0bf6 Merge pull request #1198 from efiop/master
hdfs: don't use close_fds on Windows
2018-10-09 21:00:55 +03:00
Ruslan Kuprieiev 971f3259f1 hdfs: don't use close_fds on Windows
Fixes #1197

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-09 20:42:18 +03:00
Ruslan Kuprieiev a4f9789aa4 travis: upload coverage on python 2 2018-10-08 03:30:02 +03:00
Ruslan Kuprieiev 584972ba0a travis: upload coverage on every build 2018-10-08 03:08:35 +03:00
Ruslan Kuprieiev b2b76b84d6 Merge pull request #1195 from efiop/master
dvc: stop using ntfsutils module
2018-10-07 18:07:51 +03:00
Ruslan Kuprieiev a6ea2f3db9 dvc: stop using ntfsutils module
Fixes #1162

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-07 17:49:29 +03:00
Ruslan Kuprieiev 881f188016 tests: run in parallel mode 2018-10-07 03:15:56 +03:00
Ruslan Kuprieiev c3f145591b Merge pull request #1193 from efiop/master
test: upload coverage only on CRON
2018-10-07 02:19:27 +03:00
Ruslan Kuprieiev 1a8842d8c6 test: upload coverage only on CRON
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-07 02:09:46 +03:00
Ruslan Kuprieiev 143a7f79a1 test: add version requirements for codecov and coverage 2018-10-06 23:24:55 +03:00
Ruslan Kuprieiev 7bff8c6f89 dagascii: don't try to cover tty code 2018-10-06 16:46:01 +03:00
Ruslan Kuprieiev 2daa694bf2 Merge pull request #1192 from efiop/master
dagascii: add more keys
2018-10-06 15:52:52 +03:00
Ruslan Kuprieiev 27a96ef814 dagascii: add more keys
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-06 15:41:36 +03:00
Ruslan Kuprieiev 315bbfde3b Merge pull request #1191 from efiop/master
show: use asciimatics to provide scrollable graph
2018-10-06 14:31:43 +03:00
Ruslan Kuprieiev 53b37bb266 show: use asciimatics to provide scrollable graph
Fixes #1177

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-06 01:04:00 +03:00
Ruslan Kuprieiev 005db9bb3a Merge pull request #1190 from efiop/master
dvc: add a note about dvc.org/support to error msgs
2018-10-05 22:24:20 +03:00
Ruslan Kuprieiev 7b36fd007c dvc: add a note about dvc.org/support to error msgs
Fixes #1152

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-05 21:26:55 +03:00
Ruslan Kuprieiev 2c9b08a89b Merge pull request #1189 from mroutis/update-networkx
Update networkx package (== 2.1, >= 2.1)
2018-10-05 03:07:54 +03:00
Ruslan Kuprieiev 66a962976e Merge pull request #1188 from mroutis/add-aur-repo
Add AUR package
2018-10-05 03:07:28 +03:00
Mr. Outis 4eff31dd5e Update README noticing that the AUR package is not officially maintained 2018-10-04 18:55:26 -05:00
Mr. Outis 377be75dd0 Update networkx package in setup.py 2018-10-04 18:44:10 -05:00
Ruslan Kuprieiev e015b9a167 readme: add a new line 2018-10-05 02:14:12 +03:00
Mr. Outis 2a7ec2d2df Update networkx package (== 2.1, >= 2.1)
The latest version is 2.2 (https://pypi.org/project/networkx).

Updating it to prevent collisions with some package managers already
using the latest one.
2018-10-04 18:08:27 -05:00
Mr. Outis 3d34f4e8c7 Add arch linux repository 2018-10-04 18:03:56 -05:00
Ruslan Kuprieiev c10bd344ea Merge pull request #1185 from efiop/master
stage: properly report StageFileFormatError
2018-10-04 13:30:11 +03:00
Ruslan Kuprieiev 6a8cd31ac9 stage: properly report StageFileFormatError
Fixes #1186

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-04 13:21:01 +03:00
Ruslan Kuprieiev 2c49d7d004 Merge pull request #1182 from efiop/master
gc: add a comfirmation prompt
2018-10-03 00:00:15 +03:00
Ruslan Kuprieiev 990bbe32c1 gc: add a comfirmation prompt
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-02 23:39:45 +03:00
Ruslan Kuprieiev f3668b0a4e cli: status: fix typo 2018-10-02 01:41:45 +03:00
Ruslan Kuprieiev 73eaa2a611 dvc: bump to 0.19.7 2018-10-01 21:35:42 +03:00
Ruslan Kuprieiev e63ae3d676 Merge pull request #1181 from jeepkd/fix-aws-env
s3: fix dvc not loading AWS settings from environment variables
2018-10-01 21:32:22 +03:00
Jeep Kiddee 9528ad6a1d Fix dvc not loading AWS settings from environment variables 2018-10-02 00:59:59 +07:00
Ruslan Kuprieiev aea92e0bdf Merge pull request #1180 from efiop/master
Misc test improvements
2018-10-01 18:26:42 +03:00
Ruslan Kuprieiev d1077c4457 test: improve coverage
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-01 18:11:17 +03:00
Ruslan Kuprieiev edf73dc52f test: skip hdfs test if can't connect
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-01 16:29:55 +03:00
Ruslan Kuprieiev b587f7833a dvc: bump to 0.19.6 2018-10-01 15:47:20 +03:00
Ruslan Kuprieiev 998918c690 Merge pull request #1179 from efiop/master
Misc fixes
2018-10-01 15:44:54 +03:00
Ruslan Kuprieiev 8aa53a5e71 dvc: introduce per-remote --jobs default
Fixes #1167

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-01 15:28:38 +03:00
Ruslan Kuprieiev cc2ebb2099 dvc: introduce -T|--all-tags option
Fixes #1150

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-01 12:13:17 +03:00
Ruslan Kuprieiev b9439f2e5b dvc: bump to 0.19.5 2018-09-29 23:06:51 +03:00
Ruslan Kuprieiev d112d2a829 Merge pull request #1175 from efiop/master
Misc fixes
2018-09-28 21:55:58 +03:00
Ruslan Kuprieiev f4fc3f47e4 show: don't use asciicanvas package
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 21:42:59 +03:00
Ruslan Kuprieiev 417ad8706e state: use state version
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 21:42:59 +03:00
Ruslan Kuprieiev 2e09dbb592 output: be more verbose
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 21:42:59 +03:00
Ruslan Kuprieiev 67d63a4a5b updater: check for updates every day
Our release schedule got really quick, so we can't afford to wait a week between checking for updates, because there might be new important versions with bugfixes released every day. Checking once per day should not cause too much inconvenience for users.
2018-09-28 18:20:29 +03:00
Ruslan Kuprieiev 8eb6fd5285 Merge pull request #1173 from efiop/master
repro: introduce -P|--all-pipelines
2018-09-28 13:58:48 +03:00
Ruslan Kuprieiev 669b2ec303 repro: introduce -P|--all-pipelines
Fixes #1165

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 13:33:23 +03:00
Ruslan Kuprieiev 04b3cd40e0 Merge pull request #1172 from efiop/master
s3: support env variables
2018-09-28 12:26:55 +03:00
Ruslan Kuprieiev 12d6292ed6 s3: support env variables
Fixes #1160

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 12:17:00 +03:00
Ruslan Kuprieiev 27edc4f027 dvc: bump to 0.19.4 2018-09-28 11:05:52 +03:00
Ruslan Kuprieiev 5b0abe61ab Merge pull request #1171 from efiop/master
move: use relpath intead of rel_path property
2018-09-28 10:25:09 +03:00
Ruslan Kuprieiev 1e1daeaae1 move: use relpath intead of rel_path property
Fixes #1170

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-28 10:06:51 +03:00
Ruslan Kuprieiev ece2b8ae0b dvc: bump to 0.19.3 2018-09-27 08:58:36 +03:00
Ruslan Kuprieiev fcea9dccc5 Merge pull request #1166 from efiop/master
dvc: remove unused module
2018-09-26 16:26:45 +03:00
Ruslan Kuprieiev 51b390146a dvc: remove unused module
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-26 15:34:26 +03:00
Ruslan Kuprieiev 1a6dedfe60 cli: add a note that remove --outs is default behavior. 2018-09-25 09:07:32 +03:00
Ruslan Kuprieiev 88f0913f91 Merge pull request #1161 from peter-schmidbauer/patch-1
dvc: project: Fix typos in warnings
2018-09-24 12:28:10 +03:00
Peter Schmidbauer b38112795a dvc: project: Fix typos in warnings 2018-09-24 09:30:07 +02:00
Ruslan Kuprieiev 363db9125f pull: add dummy call to try establishing the connection (#1159)
Fixes #1151

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-24 04:41:55 +03:00
pared 89e03c8606 pipeline show add --dot option (#1141)
* pipeline show add --dot option

* refactor common code, pydot in requirements

* code formatting

* code formatting
2018-09-24 02:50:41 +03:00
Ruslan Kuprieiev 7bed7213dc Merge pull request #1158 from efiop/master
osx: reflink: check if clib has clonefile()
2018-09-23 14:46:36 +03:00
Ruslan Kuprieiev 76c2f812f0 osx: reflink: check if clib has clonefile()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 14:39:09 +03:00
Ruslan Kuprieiev 8d151273c8 Merge pull request #1157 from efiop/master
reflink: osx: check that not older than High Sierra
2018-09-23 13:44:27 +03:00
Ruslan Kuprieiev 10e10f9535 reflink: osx: check that not older than High Sierra
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 13:29:18 +03:00
Ruslan Kuprieiev 3fd5603921 Merge pull request #1156 from efiop/master
dvc: use ctypes instead of cffi
2018-09-23 12:53:52 +03:00
Ruslan Kuprieiev ff63376d49 dvc: use ctypes instead of cffi
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 12:45:16 +03:00
Ruslan Kuprieiev 5071b5026d setup: exclude tests 2018-09-23 09:56:19 +03:00
Ruslan Kuprieiev c08804eddd dvc: bump to 0.19.2 2018-09-23 08:59:14 +03:00
Ruslan Kuprieiev cbc8cf23f9 Merge pull request #1155 from efiop/master
requirements: require newer google-cloud
2018-09-23 08:51:16 +03:00
Ruslan Kuprieiev 5279b1ae8f requirements: stop using obsoleted google-cloud
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 08:36:42 +03:00
Ruslan Kuprieiev 057663696a Merge pull request #1154 from efiop/master
setup: misc fixes
2018-09-23 06:23:46 +03:00
Ruslan Kuprieiev 8b513fb336 setup: misc fixes
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 06:06:01 +03:00
Ivan Shcheklein 04373e81e8 update README links 2018-09-22 10:30:44 -07:00
Ivan Shcheklein 2601766323 update README links, description 2018-09-22 10:28:42 -07:00
Ruslan Kuprieiev ad216acce4 travis: osx: modify CPPFLAGS and LDFLAGS 2018-09-22 14:53:30 +03:00
Ruslan Kuprieiev 5753944161 travis: osx: install openssl 2018-09-22 13:40:25 +03:00
Ruslan Kuprieiev b7ac347c06 travis: install: upgrade setuptools 2018-09-22 13:02:40 +03:00
Ruslan Kuprieiev beedab77f1 dvc: bump to 0.19.1 2018-09-22 12:27:59 +03:00
Ruslan Kuprieiev b1b0e3413d Merge pull request #1148 from efiop/master
Build osx packages on older versions
2018-09-22 12:24:57 +03:00
Ruslan Kuprieiev c4593cd369 travis: build on older xcode version
Fixes #1147

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-22 12:12:45 +03:00
Ruslan Kuprieiev 6a55d1502f dvc: avoid using pyreflink module
Fixes #1149

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-22 12:11:18 +03:00
Ruslan Kuprieiev e93a87f451 dvc: bump to 0.19.0 2018-09-21 11:09:00 +03:00
Ruslan Kuprieiev 3fff8d771d Merge pull request #1145 from efiop/master
dvc: properly join paths for push/pull progress bar
2018-09-21 09:31:54 +03:00
Ruslan Kuprieiev 69bb753980 dvc: properly join paths for push/pull progress bar
Fixes #1095

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-21 09:22:06 +03:00
Ruslan Kuprieiev 1d654d0444 Merge pull request #1143 from efiop/master
dvc: introduce -d|--with-deps option
2018-09-21 09:14:22 +03:00
Ruslan Kuprieiev 0618d9d0dc dvc: introduce -d|--with-deps option
Fixes #1077

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-21 08:45:09 +03:00
Ruslan Kuprieiev dea130cc61 Merge pull request #1142 from jeepkd/support-git-submodule
scm: support git submodule
2018-09-21 00:24:32 +03:00
Jeep Kiddee bbd5096de7 scm: support git submodule
Fixes #1093
2018-09-21 02:17:28 +07:00
Ruslan Kuprieiev 545ea1efa5 Merge pull request #1139 from efiop/master
scm: require abs path
2018-09-20 00:48:19 +03:00
Ruslan Kuprieiev e1841f07cb scm: require abs path
Fixes #1136

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-20 00:14:16 +03:00
Ruslan Kuprieiev 04efd1d6d3 Merge pull request #1138 from efiop/master
test: don't forget to check return code
2018-09-19 23:49:11 +03:00
Ruslan Kuprieiev 89aa2a957e test: don't forget to check return code
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-19 23:38:58 +03:00
Ruslan Kuprieiev 1da4546515 Merge pull request #1137 from efiop/master
local: use is_local property
2018-09-19 22:01:46 +03:00
Ruslan Kuprieiev 49c22cb139 local: use is_local property
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-19 21:40:00 +03:00
Ruslan Kuprieiev 4aa26a0a3a Merge pull request #1135 from efiop/master
ssh: properly join chunks on python3
2018-09-19 19:31:41 +03:00
Ruslan Kuprieiev 8d5675a92c ssh: properly join chunks on python3
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-19 18:58:36 +03:00
Ruslan Kuprieiev 2b1ea1e4b9 Merge pull request #1133 from efiop/master
repro: introduce -p|--pipeline option
2018-09-19 10:01:13 +03:00
Ruslan Kuprieiev 34faad3e96 repro: introduce -p|--pipeline option
Fixes #1107

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-19 09:35:06 +03:00
Ruslan Kuprieiev 9dab3f2e49 ssh: fix order 2018-09-19 07:19:02 +03:00
Ruslan Kuprieiev 72e8b6983b Merge pull request #1132 from efiop/master
ssh: add 'password' and 'ask_password' options
2018-09-19 06:52:16 +03:00
Ruslan Kuprieiev 682737e4a2 ssh: add 'password' and 'ask_password' options
Fixes #1124

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-19 05:50:06 +03:00
Ruslan Kuprieiev 8932254c12 Merge pull request #1131 from efiop/master
dvc: list changed files
2018-09-18 13:14:39 +03:00
Ruslan Kuprieiev 0c97e19d5c dvc: list files changed
Fixes #1130

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-18 12:37:55 +03:00
Ruslan Kuprieiev e698703969 Merge pull request #1129 from efiop/master
run: check that -f doesn't contain any subdirs
2018-09-18 07:59:04 +03:00
Ruslan Kuprieiev 571713484b run: check that -f doesn't contain any subdirs
Fixes #1110

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-18 07:46:28 +03:00
Ruslan Kuprieiev 65a9370281 Merge pull request #1128 from efiop/master
state: limit number of entries
2018-09-18 03:53:18 +03:00
Ruslan Kuprieiev 0043bd5de3 state: limit number of rows
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-18 03:40:41 +03:00
Ruslan Kuprieiev 33d04d0d59 state: count number of inserts and track size of the db
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-18 01:33:31 +03:00
Ruslan Kuprieiev 5e3d7c6c2b Merge pull request #1126 from efiop/master
state optimization
2018-09-17 10:59:32 +03:00
Ruslan Kuprieiev 59dcdb02ea state: optimize operations
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev 23d00a15b9 state: remove unused method
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev 2c3c0b08bd state: save timestamp
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev 75d7fc2206 state: merge State and LinkState classes
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev d704c41682 dvc: use INTEGER for inode
Fixes #1117

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev 428811bb71 Merge pull request #1122 from efiop/master
dvc: don't scan output directories for dvc files
2018-09-15 07:45:21 +03:00
Ruslan Kuprieiev 470825aec5 dvc: don't scan output directories for dvc files
`dvc status` for directory with 100K files sped up from ~6 sec to ~2
sec.

Fixes #1121

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-15 07:24:00 +03:00
Ruslan Kuprieiev 5fbfc135da Merge pull request #1120 from efiop/master
Misc enhancements
2018-09-15 05:08:54 +03:00
Ruslan Kuprieiev b4e62772b8 state: merge state and link state
Fixes #1091

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-15 04:54:26 +03:00
Ruslan Kuprieiev 912f0f72e5 run: require at least some arguments
Fix #1100

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-15 04:54:26 +03:00
Ruslan Kuprieiev 5730b1d088 Merge pull request #1118 from efiop/master
repro: don't check for missing dependencies when running with `--dry`
2018-09-14 00:33:45 +03:00
Ruslan Kuprieiev 5c762f316e repro: don't check for missing dependencies when running with --dry
Fixes #1113

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-14 00:23:28 +03:00
Ruslan Kuprieiev 7791dc8cfb Merge pull request #1116 from efiop/master
state: remove unused params
2018-09-13 18:24:09 +03:00
Ruslan Kuprieiev dc344056c5 state: remove unused params
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-13 18:01:23 +03:00
Ruslan Kuprieiev 87e82daad2 Merge pull request #1112 from efiop/master
state: use sqlite
2018-09-12 16:24:34 +03:00
Ruslan Kuprieiev f044a6090f state: use sqlite
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-12 15:52:34 +03:00
Ruslan Kuprieiev c68823e8d0 dvc: bump to 0.18.15 2018-09-10 15:57:24 +03:00
Ruslan Kuprieiev a17dc2504d Merge pull request #1106 from efiop/master
stage: check that deps exist before running the command
2018-09-09 23:25:04 +03:00
Ruslan Kuprieiev 537e7c75a0 stage: check that deps exist before running the command
Fixes #1101

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-09 22:59:19 +03:00
Ruslan Kuprieiev 76d6b37b1e Merge pull request #1105 from efiop/master
ssh: don't recv() before checking with recv_ready()
2018-09-09 18:59:31 +03:00
Ruslan Kuprieiev 8d13ba164e ssh: don't recv() before checking with recv_ready()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-09 18:48:57 +03:00
Ruslan Kuprieiev 559c097394 dvc: bump to 0.18.14 2018-09-08 21:39:20 +03:00
Ruslan Kuprieiev c97bfea90f Merge pull request #1103 from efiop/master
ssh fixes
2018-09-08 21:30:44 +03:00
Ruslan Kuprieiev abf395da99 ssh: properly poll output
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-08 21:14:50 +03:00
Ruslan Kuprieiev 0687c4e199 ssh: add timeout
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-08 16:17:08 +03:00
Ruslan Kuprieiev d1931e3dc0 readme: add discord link 2018-09-06 23:20:57 +03:00
Ruslan Kuprieiev 78c9d1edab dvc: bump to 0.18.13 2018-09-05 17:12:19 +03:00
Ruslan Kuprieiev 7ef8645465 push/pull/fetch: lower the thread multiplier
Fixes #1094
2018-09-05 17:11:45 +03:00
Ruslan Kuprieiev 208540c66d dvc: bump to 0.18.12 2018-09-04 19:30:28 +03:00
Ruslan Kuprieiev 216a6e5c1c Merge pull request #1089 from efiop/master
state: use json.dumps() instead of dump()
2018-09-04 19:28:54 +03:00
Ruslan Kuprieiev 8d8a94bcb7 state: use json.dumps() instead of dump()
This actually 10 times faster, because everything is stored in the
memory.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-04 19:09:25 +03:00
Ruslan Kuprieiev 1a18c4d3f1 Merge pull request #1088 from efiop/master
state: limit db size and add a file size limit
2018-09-04 18:53:32 +03:00
Ruslan Kuprieiev 211b957d17 state: limit db size and add a file size limit
Fixes #1087

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-04 18:24:19 +03:00
Ruslan Kuprieiev 4f6967a017 dvc: bump to 0.18.11 2018-09-03 00:21:23 +03:00
Ruslan Kuprieiev c1ac1b4c6e Merge pull request #1082 from efiop/master
state: don't forget to lock before dumping db
2018-09-03 00:06:11 +03:00
Ruslan Kuprieiev 022eea8214 state: don't forget to lock before dumping db
Related to #1081

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-02 23:47:50 +03:00
Ruslan Kuprieiev f1f1a97fbf Merge pull request #1080 from efiop/master
state: don't forget subdirs when computing effective mtime
2018-09-01 15:56:36 +03:00
Ruslan Kuprieiev a3aefe5f16 state: don't forget subdirs when computing effective mtime
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-01 15:41:42 +03:00
Ivan Shcheklein de3d8b528f remove Slack from README 2018-08-31 10:34:50 -07:00
Ruslan Kuprieiev 9c041a375f test: sleep to compensate for poor mtime resolution 2018-08-31 05:09:32 +03:00
Ruslan Kuprieiev 086155daa6 dvc: bump to 0.18.10 2018-08-31 04:16:52 +03:00
Ruslan Kuprieiev d47e6c39ab Merge pull request #1076 from efiop/master
remote: properly remove remote section
2018-08-31 04:15:42 +03:00
Ruslan Kuprieiev 249b60c030 remote: properly remove remote section
Fixes #1075

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-31 04:07:31 +03:00
Ruslan Kuprieiev 392cd006e3 tests: sleep() before creating a file 2018-08-30 17:27:05 +03:00
Ruslan Kuprieiev 2b0391629b readme: add slack channel 2018-08-30 13:41:16 +03:00
Ruslan Kuprieiev 3b66176f6b dvc: bump to 0.18.9 2018-08-30 00:51:45 +03:00
Ruslan Kuprieiev baa6bcd035 Merge pull request #1073 from efiop/master
run: add `-y|--yes` option to bypass prompts
2018-08-30 00:51:00 +03:00
Ruslan Kuprieiev b159ace6d7 run: add -y|--yes option to bypass prompts
Fixes #1072

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-30 00:38:28 +03:00
Ruslan Kuprieiev ffde4a6a7a Merge pull request #1071 from efiop/master
readme fixes
2018-08-29 20:53:49 +03:00
Ruslan Kuprieiev 9a95698cd9 travis: check readme formatting
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 20:34:28 +03:00
Ruslan Kuprieiev 221dbf8245 readme: fix formatting
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 20:22:25 +03:00
Ruslan Kuprieiev 3d15391c7a dvc: bump to 0.18.8 2018-08-29 19:32:55 +03:00
Ruslan Kuprieiev f6f2ebb83e Merge pull request #1070 from efiop/master
dvc: properly compute mtime of the directory
2018-08-29 19:32:29 +03:00
Ruslan Kuprieiev ac80d59704 dvc: properly compute mtime of the directory
Fixes #1067

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 19:17:50 +03:00
Ruslan Kuprieiev 3574c8954b dvc: update to 0.18.7 2018-08-29 16:01:28 +03:00
Ruslan Kuprieiev 0d78705227 Merge pull request #1069 from efiop/master
Misc
2018-08-29 16:00:45 +03:00
Ruslan Kuprieiev bf6ed76678 config: don't forget to convert port to int
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 15:50:25 +03:00
Ruslan Kuprieiev 076a2145db test: don't forget to sleep before modifying a file
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 14:54:27 +03:00
Ruslan Kuprieiev bb3cca6530 dvc: bump to 0.18.6 2018-08-29 14:02:31 +03:00
Ruslan Kuprieiev 165422a066 Merge pull request #1068 from efiop/master
misc fixes
2018-08-29 13:59:30 +03:00
Ruslan Kuprieiev 49baf38871 dvc: properly compute mtime for the directory
Fixes #1067

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 13:52:20 +03:00
Ruslan Kuprieiev 3461df1d8f config: properly merge with local config
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-29 13:52:20 +03:00
Ruslan Kuprieiev 03a3cf2f80 dvc: bump to 0.18.5 2018-08-28 13:05:36 +03:00
Ruslan Kuprieiev 94911965c4 Merge pull request #1066 from efiop/master
misc pipeline improvements
2018-08-28 12:45:06 +03:00
Ruslan Kuprieiev 55b3c47bbf pipeline: process edges and nodes separately
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-28 11:36:06 +03:00
Ruslan Kuprieiev 825da210cf dvc: use __str__ method for deps/outs
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-28 11:36:06 +03:00
Ruslan Kuprieiev ac97c4021d dvc: bump to 0.18.4 2018-08-27 17:51:09 +03:00
Ruslan Kuprieiev 213e757b52 Merge pull request #1063 from efiop/master
test: cover `pipeline show`
2018-08-27 15:05:38 +03:00
Ruslan Kuprieiev 6f0e293c9a test: cover pipeline show
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-27 14:40:21 +03:00
Ruslan Kuprieiev 7783bb900b Merge pull request #1062 from efiop/master
pipeline: fix corner case
2018-08-27 13:59:49 +03:00
Ruslan Kuprieiev bdca1ce8cc pipeline: fix corner case
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-27 13:44:16 +03:00
Ruslan Kuprieiev 80912b4a2f Merge pull request #1061 from efiop/master
ssh enhancements
2018-08-27 13:41:06 +03:00
Ruslan Kuprieiev c46bba6889 ssh: configure private key file
Fixes #1059

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-27 12:54:47 +03:00
Ruslan Kuprieiev 1283689ed4 ssh: add ability to configure port
Fixes #1060

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-27 09:31:35 +03:00
Ruslan Kuprieiev 9c5bbecb2e Merge pull request #1056 from efiop/master
repro: print a msg when pipeline is up to date
2018-08-24 20:08:29 +03:00
Ruslan Kuprieiev 316ea1d7cb repro: print a msg when pipeline is up to date
Fixes #1049

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-24 19:37:33 +03:00
Ruslan Kuprieiev 2139978646 Merge pull request #1055 from efiop/master
dvc: check stage filename
2018-08-24 19:00:52 +03:00
Ruslan Kuprieiev fa89ec8789 dvc: check stage filename
Fixes #1054

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-24 18:45:56 +03:00
Ruslan Kuprieiev 93ef937fc2 dvc: bump to 0.18.3 2018-08-24 15:54:04 +03:00
Ruslan Kuprieiev 253315e4a8 Merge pull request #1052 from efiop/master
dependency: only use rel_path for local deps
2018-08-24 14:54:39 +03:00
Ruslan Kuprieiev 9579cb7928 config: fix ConfigError
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-24 14:36:38 +03:00
Ruslan Kuprieiev 18f0ca0d92 dependency: only use rel_path for local deps
Fixes #1048

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-24 14:36:38 +03:00
Ivan Shcheklein d555bae6e1 try to use flow GIF 2018-08-23 22:57:32 -07:00
Ruslan Kuprieiev 4a43a29f11 dvc: update to 0.18.2 2018-08-23 13:14:50 +03:00
Ruslan Kuprieiev 2ff7d9b571 Merge pull request #1047 from efiop/master
s3: properly configure session and clients
2018-08-23 10:36:16 +03:00
Ruslan Kuprieiev 9d303b6b1b s3: properly configure session and clients
Fixes #1020

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-23 10:17:11 +03:00
Ruslan Kuprieiev f682e556ec Merge pull request #1046 from efiop/master
test: cover pipeline show --ascii
2018-08-23 08:07:47 +03:00
Ruslan Kuprieiev 26e007f407 test: cover pipeline show --ascii
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-23 07:57:23 +03:00
Ruslan Kuprieiev a7f14c2d3c Merge pull request #1045 from efiop/master
dvc: print command specific help
2018-08-23 07:39:37 +03:00
Ruslan Kuprieiev 79f9cedd37 dvc: print command specific help
Fixes #1044

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-23 07:30:37 +03:00
Ruslan Kuprieiev 623d3e2fed dvc: bump to 0.18.1 2018-08-23 04:48:45 +03:00
Ruslan Kuprieiev cb7eed71de setup: update grandalf reqs 2018-08-23 04:42:27 +03:00
Ruslan Kuprieiev 9135944d7c reqs: use grandalf 0.6 2018-08-23 04:41:49 +03:00
Ruslan Kuprieiev ac4fff2c45 dvc: bump to 0.18.0 2018-08-22 17:22:04 +03:00
Ruslan Kuprieiev 912538f165 Merge pull request #1040 from efiop/master
dvc: use our errors in the argparser
2018-08-22 13:05:52 +03:00
Ruslan Kuprieiev 35c5d4c294 dvc: use our errors in the argparser
Fixes #1037

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-22 10:13:58 +03:00
Ruslan Kuprieiev 3887e973c4 Merge pull request #1036 from efiop/master
logger: use colored prefix instead of coloring everything
2018-08-22 05:32:34 +03:00
Ruslan Kuprieiev 173a9fc7aa logger: use colored prefix instead of coloring everything
Fixes #1034

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-22 05:25:27 +03:00
Ruslan Kuprieiev 0281c2714c Merge pull request #1033 from efiop/master
dagascii: make vertical space smaller
2018-08-22 05:15:58 +03:00
Ruslan Kuprieiev 6d10bff23b dagascii: make vertical space smaller
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-22 05:03:48 +03:00
Ruslan Kuprieiev ff5152ee5f Merge pull request #1027 from efiop/master
dvc: add --recursive
2018-08-21 10:51:07 +03:00
Ruslan Kuprieiev c6f642fd6b dvc: add --recursive
Fixes #1026

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-21 10:41:19 +03:00
Ruslan Kuprieiev ee0dcc384d Merge pull request #1024 from efiop/master
Visualize DAG in terminal using Sugiyama layout and ASCII rendering
2018-08-19 21:17:55 +03:00
Ruslan Kuprieiev 8a484df228 pipeline: use Dvcfile by default
Fixes #970

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-19 21:01:47 +03:00
Ruslan Kuprieiev c378e4b3d2 dvc: pipeline show: create ASCII visualizations for DAG
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-19 21:01:12 +03:00
Ruslan Kuprieiev 00d53f50b1 Merge pull request #1023 from efiop/master
metrics: use --xpath if no xpath is specified in the stage file
2018-08-18 10:18:25 +03:00
Ruslan Kuprieiev 65ffc7d248 metrics: use --xpath if no xpath is specified in the stage file
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-18 05:06:29 +03:00
Ruslan Kuprieiev 95479e6d6b dvc: bump to 0.17.1 2018-08-17 14:24:42 +03:00
Ruslan Kuprieiev 79b9ae75e9 Merge pull request #1021 from efiop/master
dvc: properly resolve dvc name for external local files
2018-08-17 14:23:09 +03:00
Ruslan Kuprieiev a46b1be925 test: add test for dvc add for external local files
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-17 14:07:55 +03:00
Ruslan Kuprieiev a566e92ce0 dvc: properly resolve dvc name for external local files
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-17 11:20:59 +03:00
Ruslan Kuprieiev adbfbfd732 readme: add shell signs 2018-08-17 02:14:54 +03:00
Ruslan Kuprieiev d2b562e25a readme: fix "mailing list" section 2018-08-17 01:48:17 +03:00
Ruslan Kuprieiev bcea7a132a readme: fix header 2018-08-17 01:45:41 +03:00
Ruslan Kuprieiev d1aa1f3491 readme: wording 2018-08-17 01:32:45 +03:00
Ruslan Kuprieiev 246b80d6a4 readme: fix link 2018-08-17 01:31:33 +03:00
Ruslan Kuprieiev 8f2d5c4195 readme: add a 'Mailing list' link to the header 2018-08-17 01:27:22 +03:00
Ruslan Kuprieiev bd9da20e20 readme: add a note about our mailing list 2018-08-17 01:25:07 +03:00
Ruslan Kuprieiev c6138729f9 Merge pull request #1017 from efiop/master
help: add a better description for `dvc init -f`
2018-08-14 20:56:23 +03:00
Ruslan Kuprieiev 6743b36894 help: add a better description for dvc init -f
https://github.com/iterative/dvc.org/issues/71

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-14 20:08:28 +03:00
Ruslan Kuprieiev 0dd995453f dvc: bump to 0.17.0 2018-08-14 05:12:53 +03:00
Dmitry Petrov 42a15d91f7 Merge pull request #1016 from iterative/issue_templates
Simplify issue templates
2018-08-13 19:05:23 -07:00
Dmitry a6cb238982 Simplify issue templates 2018-08-13 18:49:06 -07:00
Ruslan Kuprieiev 1b4d7abce3 Update .coveragerc 2018-08-14 03:45:34 +03:00
Ruslan Kuprieiev e6def1287d Merge pull request #1014 from efiop/master
test: improve test coverage
2018-08-14 02:41:19 +03:00
Ruslan Kuprieiev ce5a46d032 test: test output duplication
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-14 02:33:25 +03:00
Ruslan Kuprieiev fcadd18069 coverage: ignore code dependent on tty
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-14 02:33:25 +03:00
Ruslan Kuprieiev a3588c4504 Merge pull request #1013 from efiop/master
Misc repro fixes
2018-08-14 02:16:14 +03:00
Ruslan Kuprieiev 86322fbe35 repro: print help when Dvcfile doesn't exist
Fixes #1003

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-14 02:08:12 +03:00
Ruslan Kuprieiev c635adbfc7 repro: fix msgs
Fixes #1008

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-14 02:02:07 +03:00
Ruslan Kuprieiev 6e95508e9e Merge pull request #1012 from efiop/master
destroy: mention removing data files
2018-08-13 08:08:30 +03:00
Ruslan Kuprieiev aaca0e7a38 destroy: mention removing data files
Fixes #1000

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-13 07:30:58 +03:00
Ruslan Kuprieiev 487993995a Merge pull request #1010 from efiop/master
misc fixes
2018-08-13 07:16:06 +03:00
Ruslan Kuprieiev e84f231c34 help: add cmd description
Fixes #1007

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-13 07:02:40 +03:00
Ruslan Kuprieiev 2b19b6bbfa prompt: add y/n to msg
Fixes #999

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-13 06:29:05 +03:00
Ruslan Kuprieiev db017543d5 dvc: detect output duplication
Fixes #1006

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-13 06:24:32 +03:00
Ruslan Kuprieiev bc2c84370d run: change prompt message
Fixes #1005

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-13 05:43:56 +03:00
Ruslan Kuprieiev d34119d6f0 Merge pull request #1004 from efiop/master
status: print msg if there is nothing to reproduce
2018-08-13 05:31:33 +03:00
Ruslan Kuprieiev b940380e12 status: print msg if there is nothing to reproduce
Fixes #1001

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-13 04:52:19 +03:00
Ruslan Kuprieiev e3d39063ba Merge pull request #1002 from efiop/master
help: fix punctuation
2018-08-13 04:20:54 +03:00
Ruslan Kuprieiev 11db27f333 help: fix punctuation
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-13 03:43:13 +03:00
Ruslan Kuprieiev 15210f73b2 Merge pull request #998 from efiop/master
travis: retry pip install
2018-08-13 00:17:30 +03:00
Ruslan Kuprieiev 9be3f4f122 travis: retry pip install
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-12 23:38:34 +03:00
Ruslan Kuprieiev ddf1839d38 Merge pull request #997 from efiop/master
Misc fixes
2018-08-11 21:39:31 +03:00
Ruslan Kuprieiev 3ec0a5c4ae remote: add additional checks and msgs for checkout
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-11 21:27:07 +03:00
Ruslan Kuprieiev 3814e0da4e remote: add changed() method
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-11 05:37:08 +03:00
Ruslan Kuprieiev 292702473b remote: local: rename changed() -> changed_cache()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-11 05:37:08 +03:00
Ruslan Kuprieiev 6776f6e21c dvc: print more info on checkout
Fixes #987

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-11 05:37:08 +03:00
Ruslan Kuprieiev b261554818 dvc: remote: format "No Remote repository" error
Fixes #988

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-11 05:37:08 +03:00
Ruslan Kuprieiev 10dd535176 travis: cache pip
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-11 05:37:08 +03:00
Ruslan Kuprieiev e8949c1a77 Merge pull request #996 from efiop/master
config: add core.interactive
2018-08-10 23:49:27 +03:00
Ruslan Kuprieiev 6a8703e54e config: add core.interactive
Fixes #989

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-10 23:07:46 +03:00
Ruslan Kuprieiev 6cc0c30937 Merge pull request #994 from efiop/master
dvc: print a msg when there is no cache to remove
2018-08-10 19:30:49 +03:00
Ruslan Kuprieiev b15f5ffa3b dvc: print a msg when there is no cache to remove
Fix #990

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-10 18:52:35 +03:00
Ivan Shcheklein d343f8562c add alts to contributor images, it's better to have them if image is not loading 2018-08-09 19:14:19 -07:00
Ivan Shcheklein 321c1d5f8b Merge pull request #993 from sergey48k/master
Added hall-of-fame to README
2018-08-09 19:08:14 -07:00
Sergey Surkov f6e20b162c Addressed comments for hall-of-fame
replaced: dataversioncontrol -> iterative
replaced: sergey48K -> efiop

Note that this change means that GitHub API for hall-of-fame will be counted against efiop's limit. Should be small though: 10-20 requests per hour.
2018-08-09 19:06:15 -07:00
Sergey Surkov a249df3f5d Added hall-of-fame to README 2018-08-09 17:02:56 -07:00
Ruslan Kuprieiev fb4b979183 Merge pull request #991 from efiop/master
dvc: ignore unraleted numpy runtime warnings
2018-08-09 08:06:00 +03:00
Ruslan Kuprieiev 98a7150c65 dvc: ignore unraleted numpy runtime warnings
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-09 07:12:56 +03:00
Ruslan Kuprieiev dd99a09135 github: add "Feature request" issue template 2018-08-08 23:36:32 +03:00
Ruslan Kuprieiev f854f5a97a github: Add "Bug report" issue template 2018-08-08 23:35:04 +03:00
Ruslan Kuprieiev 9e068234ba dvc: bump to 0.16.6 2018-08-07 06:02:08 +03:00
Ruslan Kuprieiev 6600ccf478 cli: use sys.exit 2018-08-07 06:01:43 +03:00
Ruslan Kuprieiev 84abef7ac0 Merge pull request #986 from efiop/master
setup: use extras_require for backward compatibility
2018-08-07 05:32:41 +03:00
Ruslan Kuprieiev 9ff2503a72 dvc: bump to 0.16.5
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-07 05:14:34 +03:00
Ruslan Kuprieiev 4ff4a1042e setup: use extras_require for backward compatibility
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-07 05:13:35 +03:00
Ruslan Kuprieiev 2195ff2b13 dvc: bump to 0.16.4 2018-08-07 04:50:36 +03:00
Ruslan Kuprieiev e539e74fde Merge pull request #985 from efiop/master
setup: use proper env marker for futures dependency
2018-08-07 04:49:55 +03:00
Ruslan Kuprieiev c345552460 setup: use proper env marker for futures dependency
Fixes #968

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-07 04:35:13 +03:00
Ruslan Kuprieiev cee8e1ec9f Merge pull request #984 from efiop/master
scm: fix LD_LIBRARY_PATH if running from a bundle
2018-08-07 04:35:04 +03:00
Ruslan Kuprieiev e4b8cf2e0e scm: fix LD_LIBRARY_PATH if running from a bundle
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-07 04:27:35 +03:00
Ruslan Kuprieiev e2e7649afe Merge pull request #983 from efiop/master
dvc: fix LD_LIBRARY_PATH on binaries
2018-08-07 03:30:15 +03:00
Ruslan Kuprieiev 77b2e45a64 dvc: fix LD_LIBRARY_PATH on binaries
Fixes #965

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-07 03:22:16 +03:00
Ruslan Kuprieiev e403568afe Merge pull request #982 from efiop/master
dvc: print more info for add/run/checkout
2018-08-07 01:08:36 +03:00
Ruslan Kuprieiev 79eac34163 dvc: print more info for add/run/checkout
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-07 00:59:17 +03:00
Ruslan Kuprieiev d9b7f7b97a Merge pull request #981 from efiop/master
dvc: print help if CLI args are wrong
2018-08-06 20:00:40 +03:00
Ruslan Kuprieiev cbe9fde7a6 dvc: print help if CLI args are wrong
Fixes #979

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-06 19:49:31 +03:00
Ruslan Kuprieiev 24be60bf39 setup.py: fix typo 2018-08-06 00:31:27 +03:00
Ruslan Kuprieiev fe38b2768f setup: don't forget to import sys 2018-08-06 00:11:05 +03:00
Ruslan Kuprieiev 105d8fc9b5 setup: fix install requirement 2018-08-06 00:02:35 +03:00
Ruslan Kuprieiev 9d7581c6bd dvc: bump to 0.16.3 2018-08-05 23:21:09 +03:00
Ruslan Kuprieiev a14946d0bb travis: script: fix version format
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-05 23:19:01 +03:00
Ruslan Kuprieiev 6ad361b741 dvc: use custom VersionAction
On some Python versions action=version prints to stderr and on some to
stdout.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-05 23:17:36 +03:00
Ruslan Kuprieiev 6fe6c04f6c travis: script: strip new line 2018-08-05 21:22:19 +03:00
Ruslan Kuprieiev 23b167f5d7 travis: script.sh: use echo instead of print 2018-08-05 20:26:46 +03:00
Ruslan Kuprieiev 2dde6e848b travis: properly check for command existance 2018-08-05 18:43:59 +03:00
Ruslan Kuprieiev 5e5705ae7e Merge pull request #980 from efiop/master
travis: test version reported by pip
2018-08-05 18:19:01 +03:00
Ruslan Kuprieiev a2d3ced7f1 travis: test version reported by pip
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-05 18:11:00 +03:00
Ruslan Kuprieiev 0804f84679 dvc: bump to 0.16.2
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-05 17:34:16 +03:00
Ruslan Kuprieiev 02da5eabdc dvc: bump to 0.16.0 2018-08-05 04:47:23 +03:00
Ruslan Kuprieiev 85ab6a141b Merge pull request #978 from efiop/master
dvc: fix help output
2018-08-05 04:45:02 +03:00
Ruslan Kuprieiev 8d1e18d543 dvc: fix help output
Fixes #934

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-05 04:36:55 +03:00
Ruslan Kuprieiev d42809abd3 Merge pull request #977 from efiop/master
repro: introduce --dry and --interactive
2018-08-05 01:05:22 +03:00
Ruslan Kuprieiev f5bb3e4501 tests: use shallow=False with filecmp.cmp()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-05 00:52:29 +03:00
Ruslan Kuprieiev aab04a22ad repro: introduce --prompt
Fixes #894

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-05 00:21:35 +03:00
Ruslan Kuprieiev ed8052c229 repro: introduce --dry option
Fixes #972
Related to #894

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-05 00:21:35 +03:00
Ruslan Kuprieiev 7baf368367 Merge pull request #976 from efiop/master
test: improve coverage
2018-08-04 20:10:55 +03:00
Ruslan Kuprieiev b5b8b8d06c test: test for unsupported outs/deps
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-04 19:59:23 +03:00
Ruslan Kuprieiev 75a6352d3d logger: don't cover tty code
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-04 19:59:23 +03:00
Ruslan Kuprieiev baac7a16bd lock: remove unused method init()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-04 19:59:23 +03:00
Ruslan Kuprieiev e4137542df coverage: skip service lines
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-04 19:59:23 +03:00
Ruslan Kuprieiev b9a5bde22f test: cover status
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-04 19:59:23 +03:00
Ruslan Kuprieiev 891ec0bad4 test: cover metrics
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-04 19:59:23 +03:00
Ruslan Kuprieiev a0a138c69f Merge pull request #975 from efiop/master
run: ask for confirmation when overwriting stage file
2018-08-04 19:15:55 +03:00
Ivan Shcheklein 1e4f6e9f09 Merge branch 'master' of github.com:iterative/dvc 2018-08-04 09:02:49 -07:00
Ivan Shcheklein 9f5ba8c38f minor README fixes - typos, layouts, etc 2018-08-04 09:02:10 -07:00
Ruslan Kuprieiev ef7aa024f9 run: ask for confirmation when overwriting stage file
Fixes #974

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-04 18:44:13 +03:00
Ruslan Kuprieiev 48c9916937 Merge pull request #971 from sotte/fix_typos_in_cli
Add missing whitespaces in CLI help texts
2018-08-04 16:02:42 +03:00
Stefan Otte 04f0320126 Add missing whitespaces in CLI help texts 2018-08-04 10:18:07 +02:00
Ivan Shcheklein 536f369f7e fix typo in README 2018-08-03 22:10:18 -07:00
Ivan Shcheklein 23a4b296e9 try README improvements 2018-08-03 21:59:29 -07:00
Ivan Shcheklein a4482bac3b README image 2018-08-03 21:33:35 -07:00
Ruslan Kuprieiev e5fd24f530 Update __init__.py 2018-08-03 22:43:18 +03:00
Ruslan Kuprieiev 4dc61beab0 travis: fix typo 2018-08-03 21:35:44 +03:00
Ruslan Kuprieiev f02bf73587 reqs: only requre futures pkg on python 2.7
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-03 20:55:23 +03:00
Ruslan Kuprieiev cb13e45632 dvc: bump to 0.15.2 2018-08-03 20:43:46 +03:00
Ruslan Kuprieiev d0ce57adac reqs: move pytest to test-requirements
Fixes #966

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-03 13:33:24 +03:00
Ruslan Kuprieiev eaa994527d reqs: don't forget futures
Fixes #967

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-03 13:25:36 +03:00
Ruslan Kuprieiev 89617501a4 Merge pull request #964 from efiop/master
dvc: additional progress info
2018-08-02 21:27:43 +03:00
Ruslan Kuprieiev 8be51b6056 dvc: additional progress info
Fixes #957

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-02 21:15:04 +03:00
Ivan Shcheklein f7a17a9336 Merge pull request #963 from konrad/master
fix link to tutorial
2018-08-02 09:47:00 -07:00
Konrad Förstner 2788288e8e fix link to tutorial 2018-08-02 18:39:08 +02:00
Ruslan Kuprieiev 80e7040a21 config: print a better msg for unsupported URL
Fixes #954

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-02 16:05:59 +03:00
Ruslan Kuprieiev bd4bbf1438 build: disable binary build on Python 3.7
Fixes #962

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-02 14:46:38 +03:00
Ruslan Kuprieiev 935cbc9513 readme: remove dvc logo 2018-08-02 02:25:35 +03:00
Ruslan Kuprieiev 98c5875668 dvc: bump to 0.15.1 2018-08-02 00:39:45 +03:00
Ruslan Kuprieiev 42111b6348 readme: don't use "raw" directive
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-02 00:38:08 +03:00
Ruslan Kuprieiev 3363b85367 dvc: bump to 0.15.0 2018-08-01 22:05:48 +03:00
Dmitry Petrov 83994d6ea7 Ordering in the header fix 2018-08-01 10:52:13 -07:00
Dmitry Petrov 568a8fb904 Ordering in the header 2018-08-01 10:51:12 -07:00
Ruslan Kuprieiev 47d00e80d2 Merge pull request #960 from efiop/master
dvc: print info when computing md5 of a large file
2018-08-01 16:10:55 +03:00
Ruslan Kuprieiev a9114d57a5 dvc: print info when computing md5 of a large file
Fixes #958

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-08-01 15:39:05 +03:00
Ruslan Kuprieiev 1bfcc12853 readme: use stock plucks 2018-08-01 11:03:39 +03:00
Ivan Shcheklein be817c5b84 add alts to pictures to fail gracefully in README 2018-07-31 23:55:49 -07:00
Ivan Shcheklein 1f15cf23e7 remove backlinks generation for the contents in README 2018-07-31 23:45:14 -07:00
Ivan Shcheklein 28e2f7d06f make contents header bold 2018-07-31 23:36:30 -07:00
Ivan Shcheklein 6bf853d635 try to add contents 2018-07-31 23:32:38 -07:00
Ivan Shcheklein 8120104f2e get everything back, it was broken before for a different reason 2018-07-31 23:26:06 -07:00
Ivan Shcheklein 1cb41f0a8a try to remove links to fix pypi 2018-07-31 23:23:44 -07:00
Ivan Shcheklein ad204cf915 remove raw html, pypi support is broken 2018-07-31 23:21:27 -07:00
Ivan Shcheklein ac3963878e cleanup and simplify README 2018-07-31 23:04:52 -07:00
Ivan Shcheklein dd891a8cd3 add menu links to the README top 2018-07-31 22:55:03 -07:00
Ivan Shcheklein 6c54810034 fix enumerated list in README 2018-07-31 22:41:32 -07:00
Ivan Shcheklein f23671a26a have to use raw image html, see github markup issue #295 2018-07-31 22:34:48 -07:00
Ivan Shcheklein 229983c403 try to add logo image to README 2018-07-31 22:20:14 -07:00
Ruslan Kuprieiev 6b042fb15a Merge pull request #955 from efiop/master
status: don't forget about missing cache
2018-07-30 17:39:24 +03:00
Ruslan Kuprieiev 1879d49bc2 status: don't forget about missing cache
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-30 17:27:24 +03:00
Ruslan Kuprieiev 627606d6f2 Merge pull request #952 from efiop/master
dvc: show filenames in `push/pull` output
2018-07-26 21:48:06 +03:00
Ruslan Kuprieiev 7e4f39d8eb dvc: show filenames in push/pull output
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-26 16:50:25 +03:00
Ruslan Kuprieiev c408fa7530 dvc: bump to 0.14.4 2018-07-25 20:23:22 +03:00
Ruslan Kuprieiev 0b2728c7ea Merge pull request #951 from efiop/master
travis: test python 3.6 and 3.7
2018-07-25 18:26:25 +03:00
Ruslan Kuprieiev 00c5242558 travis: test python 3.6 and 3.7
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-25 18:19:03 +03:00
Ruslan Kuprieiev 84da47e2fd Merge pull request #950 from efiop/master
Misc fixes
2018-07-25 17:36:27 +03:00
Ruslan Kuprieiev 66f1777d9c test: ssh: don't try to ask for a password
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-25 17:28:58 +03:00
Ruslan Kuprieiev 0e9774a3f8 RFC: follow flake8 rules
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-25 17:28:58 +03:00
Ruslan Kuprieiev 60f1deec47 dvc: don't forget to pass all_branches to push/pull/fetch
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-25 17:28:58 +03:00
Ruslan Kuprieiev 0d7a2d3c83 Merge pull request #947 from efiop/master
dvc: fix errors found by flake8
2018-07-25 13:42:21 +03:00
Ruslan Kuprieiev 2e543e49c7 dvc: fix errors found by flake8
Fixes #946. Kudos @cclaus for discovering!

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-25 13:22:23 +03:00
Ruslan Kuprieiev 1c371d036c Merge pull request #946 from cclauss/patch-1
Travis CI: Flake8 spots some missing imports
2018-07-25 13:22:13 +03:00
cclauss 4f89f3d19d Travis CI: Flake8 spots some missing imports 2018-07-25 12:04:23 +02:00
Ruslan Kuprieiev 6fd1e03364 dvc: bump to 0.14.3 2018-07-24 15:46:46 +03:00
Ruslan Kuprieiev df7dc6315f Merge pull request #944 from efiop/master
Bug fixes
2018-07-24 15:46:10 +03:00
Ruslan Kuprieiev 2f7a7638e2 dvc: properly resolve cwd for dvc add
Fixes #943

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-24 15:36:28 +03:00
Ruslan Kuprieiev 0cea36a2dd dvc: don't forget to convert to ospath for dir cache entries
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-24 15:36:13 +03:00
Ruslan Kuprieiev 537678def3 dvc: bump to 0.14.2 2018-07-23 22:42:22 +03:00
Ruslan Kuprieiev ee7f9666d3 readme: rephrase "contributing" 2018-07-23 18:34:43 +03:00
Ruslan Kuprieiev 48cc465696 readme: add "contributing" section 2018-07-23 18:19:57 +03:00
Ruslan Kuprieiev e308d5697f Merge pull request #940 from efiop/master
updater improvements
2018-07-23 14:46:19 +03:00
Ruslan Kuprieiev 7c2ef0c49c updater: print "Checking for updates..."
Fixes #935

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-23 14:37:31 +03:00
Ruslan Kuprieiev c8958d737f updater: set get() timeout
Fixes #937

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-23 14:37:31 +03:00
Ruslan Kuprieiev b6e721b486 Merge pull request #939 from hfchong/fix-base-find-root
Fix _find_root in base.py
2018-07-23 12:48:47 +03:00
hfchong f7a2377779 Fix _find_root in base.py 2018-07-23 17:33:06 +08:00
Ruslan Kuprieiev 2e3ce3b3db Merge pull request #933 from efiop/master
metrics: specify xpath in stage files
2018-07-23 00:45:12 +03:00
Ruslan Kuprieiev 2363057165 metrics: add '-t|--type' and '-x|--xpath' options
Fixes #914

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-23 00:34:03 +03:00
Ruslan Kuprieiev 5a0e16a473 Merge pull request #932 from efiop/master
metrics: properly handle explicitly specified file
2018-07-22 21:43:54 +03:00
Ruslan Kuprieiev 9e576cd55b metrics: properly handle explicitly specified file
Related to #865

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-22 21:34:03 +03:00
Ruslan Kuprieiev 4bbbdb334c Merge pull request #929 from efiop/master
s3: don't use Bucket()
2018-07-20 18:51:27 +03:00
Ruslan Kuprieiev 2881cd204c s3: don't use Bucket()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-20 18:42:05 +03:00
Ruslan Kuprieiev 2ee9525ba2 Merge pull request #928 from efiop/master
status: print locked warning only when stage is locked
2018-07-20 18:13:45 +03:00
Ruslan Kuprieiev 7cde8cbbca status: print locked warning only when stage is locked
Fixes #927

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-20 17:50:07 +03:00
Ruslan Kuprieiev 627412ff89 Merge pull request #926 from efiop/master
s3: don't use Object()
2018-07-20 17:49:03 +03:00
Ruslan Kuprieiev bf78d58793 s3: don't use Object()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-20 17:05:44 +03:00
Ruslan Kuprieiev 6aa6a0c208 dvc: bump to 0.14.1 2018-07-20 16:14:51 +03:00
Ruslan Kuprieiev f49eeaaa3c Merge pull request #925 from efiop/master
s3: don't forget to configure all Session/Client params
2018-07-20 16:12:23 +03:00
Ruslan Kuprieiev 4e49345ba2 s3: don't forget to configure all Session/Client params
Fixes #923

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-20 15:56:57 +03:00
Ruslan Kuprieiev 6e6ed2d4fa Merge pull request #921 from efiop/master
improve test coverage
2018-07-19 23:15:05 +03:00
Ruslan Kuprieiev c7e788874f remote: temporarily partially disable code for azure
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 23:05:41 +03:00
Ruslan Kuprieiev 6a6a4fe0b4 test: test unsupported chemes
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 23:05:41 +03:00
Ruslan Kuprieiev 07fa6442b8 test: add dvc status simple test
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 22:56:53 +03:00
Ruslan Kuprieiev 587cc08515 tests: test unsupported remote
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 22:56:53 +03:00
Ruslan Kuprieiev 2dabcd7d2f Merge pull request #920 from efiop/master
cli: add -a alias for --all-branches
2018-07-19 22:29:35 +03:00
Ruslan Kuprieiev adad65cc00 cli: add -a alias for --all-branches
Fixes #916

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 22:10:22 +03:00
Ruslan Kuprieiev 95f53a1b97 dvc: bump to 0.14.0 2018-07-19 12:43:28 +03:00
Ruslan Kuprieiev bb0903304e Merge pull request #917 from efiop/master
improve test coverage
2018-07-19 08:46:28 +03:00
Ruslan Kuprieiev e598b0b4c7 test: repro: add test for failed stage in CLI
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:36:32 +03:00
Ruslan Kuprieiev 3be898797a test: remote: add test for failed write
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:36:32 +03:00
Ruslan Kuprieiev 41ed1bca94 tests: test dvc pipeline show --outs
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:24:55 +03:00
Ruslan Kuprieiev 90781900e2 tests: test CLI dvc metrics add
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:24:55 +03:00
Ruslan Kuprieiev 94ae130b56 dvc: fix exception name
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:24:55 +03:00
Ruslan Kuprieiev e734ad1712 destroy: raise exception
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:24:55 +03:00
Ruslan Kuprieiev bc4a5d78e3 tests: data cloud: don't forget to test fetch
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:24:55 +03:00
Ruslan Kuprieiev 637fc29b79 test: config: add test for failed write
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:24:55 +03:00
Ruslan Kuprieiev 0a3a124756 test: cover "Invalid option value" for config
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:24:55 +03:00
Ruslan Kuprieiev ff51b1ae04 test: add test for local config
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:24:55 +03:00
Ruslan Kuprieiev 9557c56a7e tests: add _find_root test
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 08:24:55 +03:00
Ruslan Kuprieiev ebd8468b1c tests: test CmdBase._set_loglevel
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 07:05:28 +03:00
Ruslan Kuprieiev 2cab8e03be command: remove unused code
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 07:00:39 +03:00
Ruslan Kuprieiev 94551b3554 Merge pull request #915 from efiop/master
travis: use openjdk
2018-07-19 06:32:25 +03:00
Ruslan Kuprieiev b03dd85522 travis: use openjdk
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 06:24:53 +03:00
Ruslan Kuprieiev 3820eb359b Merge pull request #913 from efiop/master
metrics: Show a proper message if repository does not have metrics an…
2018-07-18 14:05:32 +03:00
Ruslan Kuprieiev 3f312a83c5 metrics: Show a proper message if repository does not have metrics and show is called
Fixes #905

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-18 13:58:45 +03:00
Ruslan Kuprieiev 9bf0fc89df Merge pull request #912 from efiop/master
metrics: csv support and additional type verification
2018-07-18 12:36:57 +03:00
Ruslan Kuprieiev 6c7bf390c6 metrics: check that specified path is a text file and not a directory or binary file
Fixes #911

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-18 12:25:11 +03:00
Ruslan Kuprieiev 5565ece1f8 metrics: add support for csv and hcsv
Fixes #910
Fixes #909

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-18 12:15:59 +03:00
Ruslan Kuprieiev ebecc9996b tests: use 0.3 sec as a max startup time 2018-07-18 11:39:24 +03:00
Ruslan Kuprieiev f9ba365a25 Merge pull request #906 from efiop/master
dvc: optimize imports to speedup startup time
2018-07-18 11:32:35 +03:00
Ruslan Kuprieiev 3410f8048a dvc: optimize imports to speedup startup time
Speeds up `time dvc` from 1.2 seconds to <0.2 seconds.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-18 11:20:05 +03:00
Ruslan Kuprieiev f54428d5ed test: properly check for hadoop availability
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-18 10:12:49 +03:00
Ivan Shcheklein 6e1839df90 minor: help for pipeline 2018-07-17 16:24:53 -07:00
Ivan Shcheklein 2d9b062fc4 minor: remove a temporary file in tests 2018-07-17 16:00:30 -07:00
Ruslan Kuprieiev 169db1fe29 dvc: bump to 0.13.0 2018-07-17 16:26:22 +03:00
Ruslan Kuprieiev 870d98bc18 test: fix expected number of stages 2018-07-17 15:57:31 +03:00
Ruslan Kuprieiev dc5fec6a75 test: remove unused add() call 2018-07-17 15:47:38 +03:00
Ruslan Kuprieiev 2885c00f80 Merge pull request #900 from efiop/master
remote: misc bug fixes
2018-07-17 13:14:45 +03:00
Ruslan Kuprieiev d424fc3825 remote: misc bug fixes
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-17 13:02:31 +03:00
Ruslan Kuprieiev 5cf8c8b598 Merge pull request #898 from efiop/master
import: support external urls as outputs
2018-07-17 12:11:01 +03:00
Ruslan Kuprieiev 805eb908d0 import: support external urls
Fixes #891

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-17 12:00:12 +03:00
Ruslan Kuprieiev 39fc1d2cc6 Merge pull request #899 from yustoris/feature/s3-endpoint-url-for-new-remote
Specify endpoint_url on S3 session client
2018-07-17 08:50:41 +03:00
kenji.yamauchi 88afad7202 Specify endpoint_url on S3 session client
We have to specify `endpoint_url` option for S3 Session client
along with Resource object.
2018-07-17 13:05:48 +09:00
Ruslan Kuprieiev dd06f53bc7 remote: use path_infos instead of regular paths in upload/download
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-16 20:00:53 +03:00
Ruslan Kuprieiev be187fad6e Merge pull request #897 from efiop/master
Misc enhancements
2018-07-16 17:08:13 +03:00
Ruslan Kuprieiev 1418ab81c9 pipeline: add -o|--outs option for show
Outputs output files instead of paths to DVC files.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-16 16:57:23 +03:00
Ruslan Kuprieiev 6a01578dfb status: print a warning about locked DVC files
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-16 16:48:00 +03:00
Ruslan Kuprieiev d98a166f87 Merge pull request #896 from efiop/master
dvc: support `dvc add` for external files
2018-07-16 16:42:05 +03:00
Ruslan Kuprieiev f1a28c3812 dvc: support dvc add for external files
Fixes #890

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-16 16:34:16 +03:00
Ruslan Kuprieiev 99bd586151 Merge pull request #892 from efiop/master
Misc enhancements
2018-07-16 13:53:04 +03:00
Ruslan Kuprieiev 6d120f474e dvc: handle double dvc init
Fixes #889

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-16 13:43:46 +03:00
Ruslan Kuprieiev 6d6653840e dvc: use active stages when collecting cache
Fixes #887

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-16 13:43:46 +03:00
Ruslan Kuprieiev 7fea5d579c Merge pull request #886 from efiop/master
dvc: add confirmation for `dvc destroy`
2018-07-15 19:27:30 +03:00
Ruslan Kuprieiev fef8ea4602 dvc: add confirmation for dvc destroy
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-15 19:18:01 +03:00
Ruslan Kuprieiev 586fde5f95 Merge pull request #885 from efiop/master
dvc: implement `dvc destroy`
2018-07-15 07:59:59 +03:00
Ruslan Kuprieiev 918bb53b75 dvc: implement dvc destroy
Fixes #621

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-15 07:45:25 +03:00
Ruslan Kuprieiev 64da1b126f dvc: bump to 0.12.0
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-14 18:00:05 +03:00
Ruslan Kuprieiev f7cb4afd33 Merge pull request #883 from efiop/master
add `dvc pipeline show`
2018-07-14 11:29:29 +03:00
Ruslan Kuprieiev 71d9fb44fc dvc: add dvc pipeline show
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-14 10:49:32 +03:00
Ruslan Kuprieiev 794dcc2a58 dvc: avoid collecting stages twice
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-14 10:49:32 +03:00
Ruslan Kuprieiev c636581699 test: gc: catch OSError 2018-07-12 10:41:34 +03:00
Ruslan Kuprieiev 4668c10e6e test: gs: catch WinError 2018-07-12 10:32:34 +03:00
Ruslan Kuprieiev 9be161de54 test: use IOError instead of FileNotFoundError 2018-07-12 10:25:40 +03:00
Ruslan Kuprieiev 79826b21a4 test: gcp: handle FileNotFoundError 2018-07-12 10:14:15 +03:00
Ruslan Kuprieiev e6e11ef015 Merge pull request #880 from efiop/master
test: check if we need to test RemoteAzure with _should_test_azure
2018-07-12 09:21:23 +03:00
Ruslan Kuprieiev f476143c52 test: check if we need to test RemoteAzure with _should_test_azure
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-12 09:13:07 +03:00
Ruslan Kuprieiev 3946463551 dvc: bump to 0.11.0 2018-07-12 08:31:38 +03:00
Ruslan Kuprieiev 774e635e60 Merge pull request #879 from efiop/master
remote: s3: properly delete key
2018-07-11 23:21:44 +03:00
Ruslan Kuprieiev 9fedecdf55 remote: s3: properly delete key
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-11 23:07:50 +03:00
Ruslan Kuprieiev 69f5ea3f3a Merge pull request #878 from efiop/master
gc: introduce -c|--cloud
2018-07-11 22:54:13 +03:00
Ruslan Kuprieiev d63d1db5b6 gc: introduce -c|--cloud
Fixes #876

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-11 22:27:13 +03:00
Ruslan Kuprieiev d71df35f63 gc: take into account local cache
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-11 22:27:13 +03:00
Ruslan Kuprieiev 716c554b93 remote: allow AZURE cache
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-11 22:27:13 +03:00
Ruslan Kuprieiev d24457c14f gc: choose which cache to process inside the drivers
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-11 22:27:13 +03:00
Ruslan Kuprieiev 5f8b2f3b91 Merge pull request #875 from CatalystCode/azure-integration-tests
Add integration tests for Azure Remote
2018-07-11 16:57:32 +03:00
Ruslan Kuprieiev bb5f1de79f Merge pull request #877 from efiop/master
dvc: sort dir cache by relpath
2018-07-11 16:36:13 +03:00
Clemens Wolff 8f36587805 Set up Azurite on AppVeyor 2018-07-11 09:25:45 -04:00
Ruslan Kuprieiev 5b674c910d dvc: sort dir cache by relpath
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-11 14:01:49 +03:00
Clemens Wolff 7897dd8c45 Make tests more verbose
This helps verify in the CI that the environment variables for the
integration tests are set up correctly since we can see exactly which
tests are being executed and which ones are being skipped.
2018-07-10 16:44:29 -04:00
Clemens Wolff 8a32292f46 Fix tests on Python 2 2018-07-10 16:44:28 -04:00
Clemens Wolff 97dfc88647 Flag skipped tests during test run 2018-07-10 16:44:28 -04:00
Clemens Wolff fcd0e34269 Enable Azure integration test on Travis CI
Note: the tests are run against the Azurite [1] local emulator for Azure
Blob Storage and not against a live service.

[1] https://github.com/azure/azurite
2018-07-10 16:33:53 -04:00
Clemens Wolff a438f77db2 Move to list-then-check exists implementation 2018-07-10 16:33:52 -04:00
Clemens Wolff fcbc6d115c Wire-up Azure in TestDataCloud 2018-07-10 16:33:52 -04:00
Clemens Wolff 1b1ca68b64 Fix inspections 2018-07-10 16:33:52 -04:00
Clemens Wolff 9724bd94aa Replace os.system with subprocess.check_output
It is generally recommended to use the subprocess module instead of the
os module to run subshell commands given since the former makes it
easier to comunicate with the process, has an improved security model,
etc. More information is available in [1].

Note that this change uses `check_output` instead of `check_call` to
prevent the command output from being printed to stdout.

This change also introduces two additional enhancements:
- Skip the hadoop test if the hadoop binary is not available.
- Skip the ssh test if the ssh binary is not available.

[1] https://docs.python.org/3/library/subprocess.html#replacing-os-system
2018-07-10 16:33:14 -04:00
Ruslan Kuprieiev 3e08cd503c Merge pull request #874 from efiop/master
utils: copyfile: properly join paths
2018-07-10 21:50:36 +03:00
Ruslan Kuprieiev e9377dd65d utils: copyfile: properly join paths
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-10 21:39:46 +03:00
Clemens Wolff 581132bbe0 Fix environment variable name 2018-07-10 13:54:49 -04:00
Ruslan Kuprieiev e051964a1b Merge pull request #873 from efiop/master
collect: skip if cache dir doesn't exist
2018-07-10 20:09:44 +03:00
Ruslan Kuprieiev 2aa9288f55 collect: skip if cache dir doesn't exist
Related to #872

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-10 19:22:18 +03:00
Ruslan Kuprieiev 6996d0a9e6 Merge pull request #870 from efiop/master
remote: add comments for s3/azure exists() method
2018-07-09 23:49:48 +03:00
Ruslan Kuprieiev 1f30d29b57 remote: add comments for s3/azure exists() method
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-09 23:40:56 +03:00
Ruslan Kuprieiev abee5f07bb Merge pull request #869 from CatalystCode/code-climate
Fix CodeClimate inspections
2018-07-09 23:21:02 +03:00
Clemens Wolff afd46e1a8a Fix CodeClimate inspections 2018-07-09 16:06:10 -04:00
Ruslan Kuprieiev 1bcd7408ec Merge pull request #868 from CatalystCode/azure-blob-storage-remote
Add Azure Blob Storage remote
2018-07-09 23:01:07 +03:00
Clemens Wolff 6882a781a4 Add Azure Blob Storage remote
This change adds a new cloud remote based on Azure Blob Storage [1] and
the azure-storage-blob SDK [2].

The implementation closely follows the patterns laid out by the existing
cloud remotes (AWS and GCP). Notable differences include:

- No use of a lock in the progress-bar callback and no requirement to
  look up total transfer size ahead of time: the Azure Storage SDK
  provides the total transfer bytes and current transfer offset in its
  progress callback.

- When `exists` is called, check the existence of every blob instead of
  listing the entire container. Container list operations can be
  expensive so unless the number of checked files is very large, making
  separate requests is likely going to be more efficient.

- For increased flexibility, access credentials for the Azure Storage
  backend can be provided either connection-string style via the remote
  URL or via environment variables. The Azure Storage backend can be
  emulated via Azurite [3].

[1] https://azure.microsoft.com/en-us/services/storage/blobs/
[2] https://github.com/Azure/azure-storage-python
[3] https://github.com/Azure/azurite

This change was created together with @EricSchles.
2018-07-09 14:47:40 -04:00
Ruslan Kuprieiev 50ea36e740 dvc: bump to 0.10.2 2018-07-08 01:13:21 +03:00
Ruslan Kuprieiev c7ed95f5ba Merge pull request #867 from efiop/master
metrics bugfixes
2018-07-08 01:11:03 +03:00
Ruslan Kuprieiev 5d3a009886 metrics: require ply>=3.9 to avoid bug on binary windows package
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-08 01:02:26 +03:00
Ruslan Kuprieiev 6683f13b4e metric: print error if unable to read
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-08 01:02:26 +03:00
104 changed files with 5803 additions and 1633 deletions
+8
View File
@@ -15,6 +15,8 @@ environment:
secure: 96fJ3r2i2GohbXHwnSs5N4EplQ7q8YmLpPWM0AC+f4s=
CODECOV_TOKEN:
secure: XN4jRtmGE5Bqg8pPZkwNs7kn3UEI73Rkldqc0MGsQISZBm5TNJZOPofDMc1QnUsf
AZURE_STORAGE_CONTAINER_NAME: appveyor-tests
AZURE_STORAGE_CONNECTION_STRING: DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;
matrix:
@@ -44,6 +46,9 @@ environment:
# PYTHON_ARCH: "64"
install:
- ps: Install-Product node
- npm install -g azurite
- ps: $AzuriteProcess = Start-Process azurite-blob -PassThru
- cinst wget
- cinst awscli
- cinst gcloudsdk
@@ -71,6 +76,9 @@ after_test:
- if "%APPVEYOR_REPO_TAG%"=="true" python -m dvc pull
- if "%APPVEYOR_REPO_TAG%"=="true" .\scripts\build_windows.cmd
on_finish:
- ps: Stop-Process $AzuriteProcess
artifacts:
- path: dvc*.exe
name: DVC Installer
+2
View File
@@ -7,3 +7,5 @@ exclude_lines =
if __name__ == .__main__.:
pass
raise NotImplementedError
# pragma: no cover
# pragma: no cover
+10
View File
@@ -0,0 +1,10 @@
---
name: Bug report
about: Create a report to help us improve
---
**Please provide information about your setup**
DVC version(i.e. `dvc --version`), Platform and method of installation (pip, homebrew, pkg Mac, exe (Windows), DEB(Linux), RPM(Linux))
@@ -0,0 +1,6 @@
---
name: Feature request
about: Suggest an idea for this project
---
+18 -4
View File
@@ -1,4 +1,5 @@
language: python
cache: pip
sudo: required
dist: trusty
group: deprecated-2017Q4
@@ -6,7 +7,7 @@ python:
- "2.7"
- "3.4"
- "3.5"
# - "3.6" Not supported by pyinstaller yet
- "3.6"
# Travis doesn't support testing python on Mac yet, so we need
# to workaround it by installing it directly with brew.
#
@@ -15,18 +16,24 @@ python:
# version, that causes fpm to crash.
matrix:
include:
# 3.7 needs Xenial image because of https://github.com/travis-ci/travis-ci/issues/9069
- python: "3.7"
dist: xenial
sudo: true
- if: type = cron OR tag is present
os: osx
osx_image: xcode9.3
osx_image: xcode7.3
language: ruby
rvm: 2.4.0
env:
- PYTHON_VER=2.7.10
- PYENV_ROOT="$HOME/.pyenv"
- PATH="$PYENV_ROOT/bin:$PATH"
- CPPFLAGS="-I/usr/local/opt/openssl/include $CPPFLAGS"
- LDFLAGS="-L/usr/local/opt/openssl/lib $LDFLAGS"
- if: type = cron OR tag is present
os: osx
osx_image: xcode9.3
osx_image: xcode7.3
language: ruby
rvm: 2.4.0
env:
@@ -35,7 +42,7 @@ matrix:
- PATH="$PYENV_ROOT/bin:$PATH"
- if: type = cron OR tag is present
os: osx
osx_image: xcode9.3
osx_image: xcode7.3
language: ruby
rvm: 2.4.0
env:
@@ -48,6 +55,7 @@ before_install:
eval "$(pyenv init -)";
PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install --skip-existing --keep --verbose "$PYTHON_VER";
pyenv shell "$PYTHON_VER";
brew install openssl;
brew cask install google-cloud-sdk;
fi
- which python
@@ -71,6 +79,12 @@ before_install:
fi
install:
- bash ./scripts/ci/install.sh
- pip install flake8
before_script:
# stop the build if there are any readme formatting errors
- python setup.py checkdocs
# stop the build if there are any flake8 errors
- flake8 dvc
script:
- bash ./scripts/ci/script.sh
deploy:
+122 -45
View File
@@ -1,20 +1,68 @@
.. image:: https://img.shields.io/travis/iterative/dvc/master.svg?label=Linux%20%26%20Mac%20OS
:target: https://travis-ci.org/iterative/dvc
.. image:: https://dvc.org/static/img/logo-owl-readme.png
:target: https://dvc.org
:alt: DVC logo
.. image:: https://img.shields.io/appveyor/ci/iterative/dvc/master.svg?label=Windows
`Website <https://dvc.org>`_
• `Docs <https://dvc.org/doc>`_
• `Twitter <https://twitter.com/iterativeai>`_
• `Chat (Community & Support) <https://dvc.org/chat>`_
• `Tutorial <https://dvc.org/doc/tutorial>`_
• `Mailing List <https://sweedom.us10.list-manage.com/subscribe/post?u=a08bf93caae4063c4e6a351f6&id=24c0ecc49a>`_
.. image:: https://travis-ci.org/iterative/dvc.svg?branch=master
:target: https://travis-ci.org/iterative/dvc
:alt: Travis
.. image:: https://ci.appveyor.com/api/projects/status/github/iterative/dvc?branch=master&svg=true
:target: https://ci.appveyor.com/project/iterative/dvc/branch/master
:alt: Windows Build
.. image:: https://codeclimate.com/github/iterative/dvc/badges/gpa.svg
:target: https://codeclimate.com/github/iterative/dvc
:alt: Code Climate
.. image:: https://codecov.io/gh/iterative/dvc/branch/master/graph/badge.svg
:target: https://codecov.io/gh/iterative/dvc
:alt: Codecov
Data Version Control or DVC is an open source tool for data science projects.
It helps data scientists manage their code and data together in a simple form of Git-like commands.
|
Get started
**Data Science Version Control** or **DVC** is an **open-source** tool for data science and
machine learning projects. With a simple and flexible Git-like architecture and interface it
helps data scientists:
#. manage **machine learning models** - versioning, including data sets and transformations (scripts) that were
used to generate models;
#. make projects **reproducible**;
#. make projects **shareable**;
#. manage experiments with branching and **metrics** tracking;
It aims to replace tools like Excel and Docs that are being commonly used as a knowledge repo and
a ledger for the team, ad-hoc scripts to track and move deploy different model versions, ad-hoc
data file suffixes and prefixes.
.. contents:: **Contents**
:backlinks: none
How DVC works
=============
DVC is compatible with Git for storing code and the dependency graph (DAG), but not data files cache.
To store and share data files cache DVC supports remotes - any cloud (S3, Azure, Google Cloud, etc) or any on-premise
network storage (via SSH, for example).
.. image:: https://dvc.org/static/img/flow.gif
:target: https://dvc.org/static/img/flow.gif
:alt: how_dvc_works
Quick start
===========
Please read `Get Started <https://dvc.org/doc/get-started>`_ for the full version. Common workflow commands include:
+-----------------------------------+-------------------------------------------------------------------+
| Step | Command |
+===================================+===================================================================+
@@ -36,78 +84,107 @@ Get started
| | | ``$ dvc push`` |
+-----------------------------------+-------------------------------------------------------------------+
See more in `tutorial <https://dvc.org/doc/tutorial>`_.
Installation
============
Packages
--------
Operating system dependent packages are the recommended way to install DVC.
The latest version of the packages can be found at GitHub releases page: https://github.com/iterative/dvc/releases
Python Pip
----------
DVC could be installed via the Python Package Index (PyPI).
There are three options to install DVC: ``pip``, Homebrew, or an OS-specific package:
.. code-block:: bash
pip install dvc
Homebrew (Mac OS)
-----------------
Package
-------
Formula:
^^^^^^^^
Self-contained packages for Windows, Linux, Mac are available. The latest version of the packages can be found at
GitHub `releases page <https://github.com/iterative/dvc/releases>`_.
Homebrew
--------
.. code-block:: bash
brew install iterative/homebrew-dvc/dvc
$ brew install iterative/homebrew-dvc/dvc
Cask:
^^^^^
or:
.. code-block:: bash
brew cask install iterative/homebrew-dvc/dvc
$ brew cask install iterative/homebrew-dvc/dvc
Links
=====
AUR
---
Website: https://dvc.org
*Unofficial package*, any inquiries regarding the AUR package,
`refer to the maintainer <https://github.com/mroutis/pkgbuilds>`_.
Tutorial: https://dvc.org/doc/tutorial
.. code-block:: bash
Documentation: http://dvc.org/doc
Discussion: https://discuss.dvc.org/
# https://aur.archlinux.org/packages/dvc
$ yay -S dvc
Related technologies
====================
#. `Git-annex <https://git-annex.branchable.com/>`_ - DVC uses the idea of storing the content of large files (that you
don't want to see in your Git repository) in a local key-value store and uses file hardlinks/symlinks instead of the
copying actual files.
#. `Git-annex <https://git-annex.branchable.com/>`_ - DVC uses the idea of storing the content of large files (that you don't want to see in your Git repository) in a local key-value store and uses file hardlinks/symlinks instead of the copying actual files.
#. `Git-LFS <https://git-lfs.github.com/>`_.
#. Makefile (and it's analogues). DVC tracks dependencies (DAG).
#. `Workflow Management Systems <https://en.wikipedia.org/wiki/Workflow_management_system>`_. DVC is workflow management system designed specificaly to manage machine learning experiments. DVC was built on top of Git.
#. `Git-LFS <https://git-lfs.github.com/>`_ - DVC is compatible with any remote storage (S3, Google Cloud, Azure, SSH,
etc). DVC utilizes reflinks or hardlinks to avoid copy operation on checkouts which makes much more efficient for
large data files.
DVC is compatible with Git for storing code and the dependency graph (DAG), but not data files cache.
Data files caches can be transferred separately - now data cache transfer throught AWS S3 and GCP storge are supported.
#. *Makefile* (and its analogues). DVC tracks dependencies (DAG).
How DVC works
=============
#. `Workflow Management Systems <https://en.wikipedia.org/wiki/Workflow_management_system>`_. DVC is a workflow
management system designed specifically to manage machine learning experiments. DVC is built on top of Git.
Contributing
============
Contributions are welcome! Please see our `Contributing Guide <https://dvc.org/doc/user-guide/contributing/>`_ for more
details.
.. image:: https://dvc.org/static/img/flow.png
:target: https://dvc.org/static/img/flow.png
:alt: how_dvc_works
.. image:: https://sourcerer.io/fame/efiop/iterative/dvc/images/0
:target: https://sourcerer.io/fame/efiop/iterative/dvc/links/0
:alt: 0
.. image:: https://sourcerer.io/fame/efiop/iterative/dvc/images/1
:target: https://sourcerer.io/fame/efiop/iterative/dvc/links/1
:alt: 1
.. image:: https://sourcerer.io/fame/efiop/iterative/dvc/images/2
:target: https://sourcerer.io/fame/efiop/iterative/dvc/links/2
:alt: 2
.. image:: https://sourcerer.io/fame/efiop/iterative/dvc/images/3
:target: https://sourcerer.io/fame/efiop/iterative/dvc/links/3
:alt: 3
.. image:: https://sourcerer.io/fame/efiop/iterative/dvc/images/4
:target: https://sourcerer.io/fame/efiop/iterative/dvc/links/4
:alt: 4
.. image:: https://sourcerer.io/fame/efiop/iterative/dvc/images/5
:target: https://sourcerer.io/fame/efiop/iterative/dvc/links/5
:alt: 5
.. image:: https://sourcerer.io/fame/efiop/iterative/dvc/images/6
:target: https://sourcerer.io/fame/efiop/iterative/dvc/links/6
:alt: 6
.. image:: https://sourcerer.io/fame/efiop/iterative/dvc/images/7
:target: https://sourcerer.io/fame/efiop/iterative/dvc/links/7
:alt: 7
Mailing List
============
Want to stay up to date? Want to help improve DVC by participating in our ocassional polls? Subscribe to our `mailing list <https://sweedom.us10.list-manage.com/subscribe/post?u=a08bf93caae4063c4e6a351f6&id=24c0ecc49a>`_. No spam, really low traffic.
Copyright
=========
This project is distributed under the Apache license version 2.0 (see the LICENSE file in the project root).
By submitting a pull request for this project, you agree to license your contribution under the Apache license version 2.0 to this project.
By submitting a pull request for this project, you agree to license your contribution under the Apache license version
2.0 to this project.
+18 -7
View File
@@ -4,8 +4,10 @@ DVC
Make your data science projects reproducible and shareable.
"""
import os
import warnings
VERSION_BASE = '0.10.1'
VERSION_BASE = '0.19.10'
__version__ = VERSION_BASE
PACKAGEPATH = os.path.abspath(os.path.dirname(__file__))
@@ -15,7 +17,8 @@ VERSIONPATH = os.path.join(PACKAGEPATH, 'version.py')
if os.path.exists(os.path.join(HOMEPATH, 'setup.py')):
# dvc is run directly from source without installation or
# __version__ is called from setup.py
if os.getenv('APPVEYOR_REPO_TAG', '').lower() != 'true' and os.getenv('TRAVIS_TAG', '') == '':
if os.getenv('APPVEYOR_REPO_TAG', '').lower() != 'true' \
and os.getenv('TRAVIS_TAG', '') == '':
# Dynamically update version
try:
import git
@@ -29,20 +32,28 @@ if os.path.exists(os.path.join(HOMEPATH, 'setup.py')):
# and will provide a true version of the installed dvc
with open(VERSIONPATH, 'w+') as fd:
fd.write('# AUTOGENERATED by dvc/__init__.py\n')
fd.write('version = "{}"'.format(__version__))
except:
fd.write('version = "{}"\n'.format(__version__))
except Exception: # pragma: no cover
pass
else:
else: # pragma: no cover
# Remove version.py so that it doesn't get into the release
if os.path.exists(VERSIONPATH):
os.unlink(VERSIONPATH)
else:
else: # pragma: no cover
# dvc was installed with pip or something. Hopefully we have our
# auto-generated version.py to help us provide a true version
try:
from dvc.version import version
__version__ = version
except:
except Exception:
pass
VERSION = __version__
# Ignore numpy's runtime warnings: https://github.com/numpy/numpy/pull/432.
# We don't directly import numpy, but our dependency networkx does, causing
# these warnings in some environments. Luckily these warnings are benign and
# we can simply ignore them so that they don't show up when you are using dvc.
warnings.filterwarnings("ignore", message="numpy.dtype size changed")
warnings.filterwarnings("ignore", message="numpy.ufunc size changed")
+15 -9
View File
@@ -9,16 +9,20 @@ class Cache(object):
def __init__(self, project):
from dvc.remote import Remote
self.project = project
config = project.config._config[Config.SECTION_CACHE]
local = config.get(Config.SECTION_CACHE_LOCAL, None)
if local:
sect = project.config._config[Config.SECTION_REMOTE_FMT.format(local)]
name = Config.SECTION_REMOTE_FMT.format(local)
sect = project.config._config[name]
else:
sect = {}
cache_dir = config.get(Config.SECTION_CACHE_DIR, self.CACHE_DIR)
if not os.path.isabs(cache_dir):
cache_dir = os.path.abspath(os.path.realpath(os.path.join(project.dvc_dir, cache_dir)))
cache_dir = os.path.join(project.dvc_dir, cache_dir)
cache_dir = os.path.abspath(os.path.realpath(cache_dir))
sect[Config.SECTION_REMOTE_URL] = cache_dir
t = config.get(Config.SECTION_CACHE_TYPE, None)
if t:
@@ -26,17 +30,19 @@ class Cache(object):
self.local = Remote(project, sect)
self.s3 = self._get_remote(project, config, Config.SECTION_CACHE_S3)
self.gs = self._get_remote(project, config, Config.SECTION_CACHE_GS)
self.ssh = self._get_remote(project, config, Config.SECTION_CACHE_SSH)
self.hdfs = self._get_remote(project, config, Config.SECTION_CACHE_HDFS)
self.s3 = self._get_remote(config, Config.SECTION_CACHE_S3)
self.gs = self._get_remote(config, Config.SECTION_CACHE_GS)
self.ssh = self._get_remote(config, Config.SECTION_CACHE_SSH)
self.hdfs = self._get_remote(config, Config.SECTION_CACHE_HDFS)
self.azure = self._get_remote(config, Config.SECTION_CACHE_AZURE)
def _get_remote(self, project, config, name):
def _get_remote(self, config, name):
from dvc.remote import Remote
remote = config.get(name, None)
if not remote:
return None
sect = project.config._config[Config.SECTION_REMOTE_FMT.format(remote)]
return Remote(project, sect)
name = Config.SECTION_REMOTE_FMT.format(remote)
sect = self.project.config._config[name]
return Remote(self.project, sect)
+524 -128
View File
File diff suppressed because it is too large Load Diff
+4 -3
View File
@@ -1,13 +1,14 @@
from dvc.exceptions import DvcException
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
from dvc.logger import Logger
class CmdAdd(CmdBase):
def run(self):
for target in self.args.targets:
try:
self.project.add(target)
self.project.add(target, recursive=self.args.recursive)
except DvcException as ex:
self.project.logger.error('Failed to add \'{}\''.format(target), ex)
Logger.error('Failed to add \'{}\''.format(target), ex)
return 1
return 0
@@ -1,6 +1,5 @@
import os
from dvc.project import Project
from dvc.logger import Logger
from dvc.exceptions import DvcException
from dvc.lock import LockError
@@ -8,6 +7,8 @@ from dvc.lock import LockError
class CmdBase(object):
def __init__(self, args):
from dvc.project import Project
self.project = Project(self._find_root())
self.args = args
self._set_loglevel(args)
@@ -20,11 +21,15 @@ class CmdBase(object):
Logger.be_verbose()
def _find_root(self):
from dvc.project import Project
root = os.getcwd()
while not os.path.ismount(root):
while True:
dvc_dir = os.path.join(root, Project.DVC_DIR)
if os.path.isdir(dvc_dir):
return root
if os.path.ismount(root):
break
root = os.path.dirname(root)
msg = "Not a dvc repository (checked up to mount point {})"
raise DvcException(msg.format(root))
@@ -34,7 +39,7 @@ class CmdBase(object):
with self.project.lock:
return self.run()
except LockError as ex:
self.project.logger.error('Failed to lock before running a command', ex)
Logger.error('Failed to lock before running a command', ex)
return 1
# Abstract methods that have to be implemented by any inheritance class
+1 -2
View File
@@ -1,5 +1,4 @@
import os
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
class CmdCheckout(CmdBase):
View File
-6
View File
@@ -1,6 +0,0 @@
from dvc.exceptions import DvcException
class CmdCommonError(DvcException):
def __init__(self, msg):
super(CmdCommonError, self).__init__('{}'.format(msg))
+7 -5
View File
@@ -1,14 +1,15 @@
import os
import configobj
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
from dvc.logger import Logger
from dvc.config import Config
from dvc.project import Project
class CmdConfig(CmdBase):
def __init__(self, args):
from dvc.project import Project
self.args = args
root_dir = self._find_root()
if args.local:
@@ -38,7 +39,8 @@ class CmdConfig(CmdBase):
try:
self.configobj.write()
except Exception as exc:
Logger.error("Failed to write config '{}'".format(self.configobj.filename), exc)
msg = "Failed to write config '{}'".format(self.configobj.filename)
Logger.error(msg, exc)
return 1
return 0
@@ -50,7 +52,7 @@ class CmdConfig(CmdBase):
if opt in self.configobj[section].keys():
del self.configobj[section][opt]
if len(self.configobj[section]) == 0 or opt == None:
if len(self.configobj[section]) == 0 or opt is None:
del self.configobj[section]
return self.save()
@@ -69,7 +71,7 @@ class CmdConfig(CmdBase):
def check_opt(self):
_section, _opt = self.args.name.strip().split('.', 1)
add = (self.args.value != None and self.args.unset == False)
add = (self.args.value is not None and self.args.unset is False)
section = self._get_key(self.configobj, _section, add)
+26 -6
View File
@@ -1,4 +1,4 @@
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
class CmdDataBase(CmdBase):
@@ -19,9 +19,16 @@ class CmdDataBase(CmdBase):
class CmdDataPull(CmdDataBase):
def do_run(self, target=None):
try:
self.project.pull(target=target, jobs=self.args.jobs, remote=self.args.remote)
self.project.pull(target=target,
jobs=self.args.jobs,
remote=self.args.remote,
show_checksums=self.args.show_checksums,
all_branches=self.args.all_branches,
all_tags=self.args.all_tags,
with_deps=self.args.with_deps)
except Exception as exc:
self.project.logger.error('Failed to pull data from the cloud', exc)
self.project.logger.error('Failed to pull data from the cloud',
exc)
return 1
return 0
@@ -29,7 +36,13 @@ class CmdDataPull(CmdDataBase):
class CmdDataPush(CmdDataBase):
def do_run(self, target=None):
try:
self.project.push(target=target, jobs=self.args.jobs, remote=self.args.remote)
self.project.push(target=target,
jobs=self.args.jobs,
remote=self.args.remote,
show_checksums=self.args.show_checksums,
all_branches=self.args.all_branches,
all_tags=self.args.all_tags,
with_deps=self.args.with_deps)
except Exception as exc:
self.project.logger.error('Failed to push data to the cloud', exc)
return 1
@@ -39,8 +52,15 @@ class CmdDataPush(CmdDataBase):
class CmdDataFetch(CmdDataBase):
def do_run(self, target=None):
try:
self.project.fetch(target=target, jobs=self.args.jobs, remote=self.args.remote)
self.project.fetch(target=target,
jobs=self.args.jobs,
remote=self.args.remote,
show_checksums=self.args.show_checksums,
all_branches=self.args.all_branches,
all_tags=self.args.all_tags,
with_deps=self.args.with_deps)
except Exception as exc:
self.project.logger.error('Failed to fetch data from the cloud', exc)
self.project.logger.error('Failed to fetch data from the cloud',
exc)
return 1
return 0
+22
View File
@@ -0,0 +1,22 @@
from dvc.command.base import CmdBase
from dvc.exceptions import DvcException
class CmdDestroy(CmdBase):
def run_cmd(self):
try:
msg = u'This will destroy all information about your pipelines, ' \
u'all data files, as well as cache in .dvc/cache.\n' \
u'Are you sure you want to continue?'
if not self.args.force \
and not self.project.prompt.prompt(msg, False):
msg = u'Cannot destroy without a confirmation from the ' \
u'user. Use \'-f\' to force.'
raise DvcException(msg)
self.project.destroy()
except Exception as exc:
self.project.logger.error('Failed to destroy DVC', exc)
return 1
return 0
+21 -2
View File
@@ -1,7 +1,26 @@
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
class CmdGC(CmdBase):
def run(self):
self.project.gc()
msg = 'This will remove all cache except the cache that is used in '
if not self.args.all_branches and not self.args.all_tags:
msg += 'the current git branch.'
elif self.args.all_branches and not self.args.all_tags:
msg += 'all git branches.'
elif not self.args.all_branches and self.args.all_tags:
msg += 'all git tags.'
else:
msg += 'all git branches and all git tags.'
self.project.logger.warn(msg)
msg = 'Are you sure you want to proceed?'
if not self.args.force and not self.project.prompt.prompt(msg):
return 1
self.project.gc(all_branches=self.args.all_branches,
all_tags=self.args.all_tags,
cloud=self.args.cloud,
remote=self.args.remote)
return 0
+3 -2
View File
@@ -1,5 +1,6 @@
from dvc.exceptions import DvcException
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
from dvc.logger import Logger
class CmdImport(CmdBase):
@@ -7,6 +8,6 @@ class CmdImport(CmdBase):
try:
self.project.imp(self.args.url, self.args.out)
except DvcException as ex:
self.project.logger.error('Failed to import {}', ex)
Logger.error('Failed to import {}'.format(self.args.url), ex)
return 1
return 0
+3 -2
View File
@@ -1,4 +1,3 @@
from dvc.project import Project, InitError
from dvc.logger import Logger
@@ -7,8 +6,10 @@ class CmdInit(object):
self.args = args
def run_cmd(self):
from dvc.project import Project, InitError
try:
Project.init('.', no_scm=self.args.no_scm)
Project.init('.', no_scm=self.args.no_scm, force=self.args.force)
except InitError as e:
Logger.error('Failed to initiate dvc', e)
return 1
+1 -2
View File
@@ -1,6 +1,5 @@
from dvc.project import Project
from dvc.logger import Logger
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
class CmdInstall(CmdBase):
+3 -2
View File
@@ -1,5 +1,5 @@
from dvc.exceptions import DvcException
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
class CmdLockBase(CmdBase):
@@ -8,7 +8,8 @@ class CmdLockBase(CmdBase):
try:
self.project.lock_stage(target, unlock=unlock)
except DvcException as ex:
msg = 'Failed to {}lock \'{}\''.format('un' if unlock else '', target)
msg = 'Failed to {}lock \'{}\''
msg.format('un' if unlock else '', target)
self.project.logger.error(msg, ex)
return 1
return 0
+61 -8
View File
@@ -1,23 +1,76 @@
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
from dvc.exceptions import DvcException
class CmdMetricsShow(CmdBase):
def run(self):
self.project.metrics_show(self.args.path,
json_path=self.args.json_path,
tsv_path=self.args.tsv_path,
htsv_path=self.args.htsv_path,
all_branches=self.args.all_branches)
try:
# backward compatibility
if self.args.json_path:
typ = 'json'
xpath = self.args.json_path
elif self.args.tsv_path:
typ = 'tsv'
xpath = self.args.tsv_path
elif self.args.htsv_path:
typ = 'htsv'
xpath = self.args.htsv_path
elif self.args.csv_path:
typ = 'csv'
xpath = self.args.csv_path
elif self.args.hcsv_path:
typ = 'hcsv'
xpath = self.args.hcsv_path
else:
typ = self.args.type
xpath = self.args.xpath
self.project.metrics_show(self.args.path,
typ=typ,
xpath=xpath,
all_branches=self.args.all_branches,
all_tags=self.args.all_tags)
except DvcException as exc:
self.project.logger.error('Failed to show metrics', exc)
return 1
return 0
class CmdMetricsModify(CmdBase):
def run(self):
try:
self.project.metrics_modify(self.args.path,
typ=self.args.type,
xpath=self.args.xpath)
except DvcException as exc:
self.project.logger.error('Failed to modify metrics', exc)
return 1
return 0
class CmdMetricsAdd(CmdBase):
def run(self):
self.project.metrics_add(self.args.path)
try:
self.project.metrics_add(self.args.path,
self.args.type,
self.args.xpath)
except DvcException as exc:
msg = 'Failed to add metric file \'{}\''.format(self.args.path)
self.project.logger.error(msg, exc)
return 1
return 0
class CmdMetricsRemove(CmdBase):
def run(self):
self.project.metrics_remove(self.args.path)
try:
self.project.metrics_remove(self.args.path)
except DvcException as exc:
msg = 'Failed to remove metric file \'{}\''.format(self.args.path)
self.project.logger.error(msg, exc)
return 1
return 0
+4 -3
View File
@@ -1,5 +1,5 @@
from dvc.exceptions import DvcException
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
class CmdMove(CmdBase):
@@ -7,7 +7,8 @@ class CmdMove(CmdBase):
try:
self.project.move(self.args.src, self.args.dst)
except DvcException as ex:
self.project.logger.error('Failed to move \'{}\' -> \'{}\''.format(self.args.src,
self.args.dst), ex)
msg = 'Failed to move \'{}\' -> \'{}\''.format(self.args.src,
self.args.dst)
self.project.logger.error(msg, ex)
return 1
return 0
+112
View File
@@ -0,0 +1,112 @@
import os
from dvc.exceptions import DvcException
from dvc.command.base import CmdBase
from dvc.dagascii import Dagascii
class CmdPipelineShow(CmdBase):
def _show(self, target, commands, outs):
import networkx
from dvc.stage import Stage
stage = Stage.load(self.project, target)
G = self.project.graph()[0]
stages = networkx.get_node_attributes(G, 'stage')
node = os.path.relpath(stage.path, self.project.root_dir)
for n in networkx.dfs_postorder_nodes(G, node):
if commands:
self.project.logger.info(stages[n].cmd)
elif outs:
for out in stages[n].outs:
self.project.logger.info(str(out))
else:
self.project.logger.info(n)
def __build_graph(self, target, commands, outs):
import networkx
from dvc.stage import Stage
stage = Stage.load(self.project, target)
node = os.path.relpath(stage.path, self.project.root_dir)
pipelines = list(filter(lambda g: node in g.nodes(),
self.project.pipelines()))
assert len(pipelines) == 1
G = pipelines[0]
stages = networkx.get_node_attributes(G, 'stage')
nodes = []
for n in G.nodes():
stage = stages[n]
if commands:
if stage.cmd is None:
continue
nodes.append(stage.cmd)
elif outs:
for out in stage.outs:
nodes.append(str(out))
else:
nodes.append(stage.relpath)
edges = []
for e in G.edges():
from_stage = stages[e[0]]
to_stage = stages[e[1]]
if commands:
if to_stage.cmd is None:
continue
edges.append((from_stage.cmd, to_stage.cmd))
elif outs:
for from_out in from_stage.outs:
for to_out in to_stage.outs:
edges.append((str(from_out),
str(to_out)))
else:
edges.append((from_stage.relpath, to_stage.relpath))
return nodes, edges
def _show_ascii(self, target, commands, outs):
nodes, edges = self.__build_graph(target, commands, outs)
if not nodes:
return
d = Dagascii(nodes, edges)
d.draw()
def __write_dot(self, target, commands, outs, filename):
import networkx
from networkx.drawing.nx_pydot import write_dot
_, edges = self.__build_graph(target, commands, outs)
edges = [edge[::-1] for edge in edges]
simple_g = networkx.DiGraph()
simple_g.add_edges_from(edges)
write_dot(simple_g, filename)
def run(self, unlock=False):
for target in self.args.targets:
try:
if self.args.ascii:
self._show_ascii(target,
self.args.commands,
self.args.outs)
elif self.args.dot:
self.__write_dot(target,
self.args.commands,
self.args.outs,
self.args.dot)
else:
self._show(target,
self.args.commands,
self.args.outs)
except DvcException as ex:
msg = 'Failed to show pipeline for \'{}\''.format(target)
self.project.logger.error(msg, ex)
return 1
return 0
+37 -4
View File
@@ -1,4 +1,6 @@
import os
import re
import configobj
from dvc.config import Config
from dvc.command.config import CmdConfig
@@ -13,15 +15,45 @@ class CmdRemoteAdd(CmdConfig):
return ret
if self.args.default:
Logger.info('Setting \'{}\' as a default remote.'.format(self.args.name))
ret = self.set(Config.SECTION_CORE, Config.SECTION_CORE_REMOTE, self.args.name)
msg = 'Setting \'{}\' as a default remote.'.format(self.args.name)
Logger.info(msg)
ret = self.set(Config.SECTION_CORE,
Config.SECTION_CORE_REMOTE,
self.args.name)
return ret
class CmdRemoteRemove(CmdConfig):
def _remove_default(self, config_file, remote):
path = os.path.join(os.path.dirname(self.config_file),
config_file)
config = configobj.ConfigObj(path)
core = config.get(Config.SECTION_CORE, None)
if core is None:
return
default = core.get(Config.SECTION_CORE_REMOTE, None)
if default is None:
return
if default == remote:
del config[Config.SECTION_CORE][Config.SECTION_CORE_REMOTE]
if len(config[Config.SECTION_CORE]) == 0:
del config[Config.SECTION_CORE]
config.write()
def run(self):
section = Config.SECTION_REMOTE_FMT.format(self.args.name)
return self.unset(section, Config.SECTION_REMOTE_URL)
ret = self.unset(section)
if ret != 0:
return ret
self._remove_default(Config.CONFIG, self.args.name)
self._remove_default(Config.CONFIG_LOCAL, self.args.name)
return 0
class CmdRemoteModify(CmdConfig):
@@ -37,6 +69,7 @@ class CmdRemoteList(CmdConfig):
r = re.match(Config.SECTION_REMOTE_REGEX, section)
if r:
name = r.group('name')
url = self.configobj[section].get(Config.SECTION_REMOTE_URL, '')
url = self.configobj[section].get(Config.SECTION_REMOTE_URL,
'')
Logger.info('{}\t{}'.format(name, url))
return 0
+3 -2
View File
@@ -1,5 +1,6 @@
from dvc.exceptions import DvcException
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
from dvc.logger import Logger
class CmdRemove(CmdBase):
@@ -9,6 +10,6 @@ class CmdRemove(CmdBase):
try:
self.project.remove(target, outs_only=outs_only)
except DvcException as ex:
self.project.logger.error('Failed to remove {}'.format(target), ex)
Logger.error('Failed to remove {}'.format(target), ex)
return 1
return 0
+15 -11
View File
@@ -1,8 +1,8 @@
import os
from dvc.command.common.base import CmdBase
from dvc.project import ReproductionError
from dvc.stage import Stage
from dvc.command.base import CmdBase
from dvc.command.status import CmdDataStatus
from dvc.exceptions import DvcException
class CmdRepro(CmdBase):
@@ -15,21 +15,25 @@ class CmdRepro(CmdBase):
ret = 0
for target in self.args.targets:
try:
self.project.reproduce(target,
stages = self.project.reproduce(
target,
recursive=recursive,
force=self.args.force)
force=self.args.force,
dry=self.args.dry,
interactive=self.args.interactive,
pipeline=self.args.pipeline,
all_pipelines=self.args.all_pipelines)
if len(stages) == 0:
self.project.logger.info(CmdDataStatus.UP_TO_DATE_MSG)
if self.args.metrics:
self.project.metrics_show()
except ReproductionError as ex:
except DvcException as ex:
msg = 'Failed to reproduce \'{}\''.format(target)
self.project.logger.error(msg, ex)
ret = 1
break
except Exception as ex:
msg = 'Failed to reproduce \'{}\' - unexpected error'.format(target)
self.project.logger.error(msg, ex)
ret = 1
break
os.chdir(saved_dir)
return ret
+1 -1
View File
@@ -1,6 +1,6 @@
import os
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
class CmdRoot(CmdBase):
+5 -36
View File
@@ -1,25 +1,19 @@
import os
from dvc.command.common.base import CmdBase
from dvc.logger import Logger
from dvc.stage import Stage
from dvc.command.base import CmdBase
from dvc.exceptions import DvcException
class CmdRun(CmdBase):
def run(self):
fname = self.stage_file_name(self.args.file,
self.args.outs,
self.args.outs_no_cache,
self.args.metrics_no_cache)
try:
if self.args.yes:
self.project.prompt.default = True
self.project.run(cmd=' '.join(self.args.command),
outs=self.args.outs,
outs_no_cache=self.args.outs_no_cache,
metrics_no_cache=self.args.metrics_no_cache,
deps=self.args.deps,
fname=fname,
fname=self.args.file,
cwd=self.args.cwd,
no_exec=self.args.no_exec)
except DvcException as ex:
@@ -27,28 +21,3 @@ class CmdRun(CmdBase):
return 1
return 0
@staticmethod
def stage_file_name(args_file, args_outs, args_outs_no_cache, args_metrics_no_cache):
if args_file:
return args_file
if args_outs:
result = CmdRun._stage_file_basename(args_outs[0])
elif args_outs_no_cache:
result = CmdRun._stage_file_basename(args_outs_no_cache[0])
elif args_metrics_no_cache:
result = CmdRun._stage_file_basename(args_metrics_no_cache[0])
else:
result = Stage.STAGE_FILE
Logger.info(u'Using \'{}\' as a stage file'.format(result))
return result
@staticmethod
def _stage_file_basename(fname):
result = os.path.basename(fname)
if len(result) == 0:
result = os.path.basename(os.path.dirname(fname))
result += Stage.STAGE_FILE_SUFFIX
return result
+18 -8
View File
@@ -1,16 +1,14 @@
import os
from dvc.command.data_sync import CmdDataBase
class CmdDataStatus(CmdDataBase):
STATUS_LEN = 10
STATUS_INDENT = '\t'
UP_TO_DATE_MSG = "Pipeline is up to date. Nothing to reproduce."
def _normalize(self, s):
s+=':'
if len(s) >= self.STATUS_LEN:
return s
s += ':'
assert len(s) < self.STATUS_LEN
return s + (self.STATUS_LEN - len(s))*' '
def _show(self, status, indent=0):
@@ -21,13 +19,25 @@ class CmdDataStatus(CmdDataBase):
self.project.logger.info('{}{}'.format(ind, key))
self._show(value, indent+1)
else:
self.project.logger.info('{}{}{}'.format(ind, self._normalize(value), key))
msg = '{}{}{}'.format(ind, self._normalize(value), key)
self.project.logger.info(msg)
def do_run(self, target=None):
indent = 1 if self.args.cloud else 0
try:
status = self.project.status(target=target, jobs=self.args.jobs, cloud=self.args.cloud, remote=self.args.remote)
self._show(status, indent)
st = self.project.status(target=target,
jobs=self.args.jobs,
cloud=self.args.cloud,
show_checksums=self.args.show_checksums,
remote=self.args.remote,
all_branches=self.args.all_branches,
all_tags=self.args.all_tags,
with_deps=self.args.with_deps)
if st:
self._show(st, indent)
else:
self.project.logger.info(self.UP_TO_DATE_MSG)
except Exception as exc:
self.project.logger.error('Failed to obtain data status', exc)
return 1
+103 -40
View File
@@ -2,16 +2,17 @@
DVC config objects.
"""
import os
import schema
import configobj
from schema import Schema, Optional, And, Use, Regex
from dvc.exceptions import DvcException
class ConfigError(DvcException):
""" DVC config exception """
def __init__(self, ex=None):
super(ConfigError, self).__init__('Config file error', ex)
def __init__(self, msg, ex=None):
super(ConfigError, self).__init__('Config file error: {}'.format(msg),
ex)
def supported_url(url):
@@ -28,14 +29,40 @@ def supported_cache_type(types):
return True
def supported_loglevel(level):
return level in ['info', 'debug', 'warning', 'error']
def supported_cloud(cloud):
return cloud in ['aws', 'gcp', 'local', '']
def is_bool(val):
return val.lower() in ['true', 'false']
def to_bool(val):
return val.lower() == 'true'
def is_whole(val):
return int(val) >= 0
def is_percent(val):
return int(val) >= 0 and int(val) <= 100
class Config(object):
CONFIG = 'config'
CONFIG_LOCAL = 'config.local'
SECTION_CORE = 'core'
SECTION_CORE_LOGLEVEL = 'loglevel'
SECTION_CORE_LOGLEVEL_SCHEMA = schema.And(schema.Use(str.lower), lambda l: l in ('info', 'debug', 'warning', 'error'))
SECTION_CORE_LOGLEVEL_SCHEMA = And(Use(str.lower), supported_loglevel)
SECTION_CORE_REMOTE = 'remote'
SECTION_CORE_INTERACTIVE_SCHEMA = And(str, is_bool, Use(to_bool))
SECTION_CORE_INTERACTIVE = 'interactive'
SECTION_CACHE = 'cache'
SECTION_CACHE_DIR = 'dir'
@@ -46,30 +73,36 @@ class Config(object):
SECTION_CACHE_GS = 'gs'
SECTION_CACHE_SSH = 'ssh'
SECTION_CACHE_HDFS = 'hdfs'
SECTION_CACHE_AZURE = 'azure'
SECTION_CACHE_SCHEMA = {
schema.Optional(SECTION_CACHE_LOCAL): str,
schema.Optional(SECTION_CACHE_S3): str,
schema.Optional(SECTION_CACHE_GS): str,
schema.Optional(SECTION_CACHE_HDFS): str,
schema.Optional(SECTION_CACHE_SSH): str,
Optional(SECTION_CACHE_LOCAL): str,
Optional(SECTION_CACHE_S3): str,
Optional(SECTION_CACHE_GS): str,
Optional(SECTION_CACHE_HDFS): str,
Optional(SECTION_CACHE_SSH): str,
Optional(SECTION_CACHE_AZURE): str,
# backward compatibility
schema.Optional(SECTION_CACHE_DIR, default='cache'): str,
schema.Optional(SECTION_CACHE_TYPE, default=None): SECTION_CACHE_TYPE_SCHEMA,
Optional(SECTION_CACHE_DIR, default='cache'): str,
Optional(SECTION_CACHE_TYPE, default=None): SECTION_CACHE_TYPE_SCHEMA,
}
# backward compatibility
SECTION_CORE_CLOUD = 'cloud'
SECTION_CORE_CLOUD_SCHEMA = schema.And(schema.Use(str.lower), lambda c: c in ('aws', 'gcp', 'local', ''))
SECTION_CORE_CLOUD_SCHEMA = And(Use(str.lower), supported_cloud)
SECTION_CORE_STORAGEPATH = 'storagepath'
SECTION_CORE_SCHEMA = {
schema.Optional(SECTION_CORE_LOGLEVEL, default='info'): schema.And(str, schema.Use(str.lower), SECTION_CORE_LOGLEVEL_SCHEMA),
schema.Optional(SECTION_CORE_REMOTE, default=''): schema.And(str, schema.Use(str.lower)),
Optional(SECTION_CORE_LOGLEVEL,
default='info'): And(str, Use(str.lower),
SECTION_CORE_LOGLEVEL_SCHEMA),
Optional(SECTION_CORE_REMOTE, default=''): And(str, Use(str.lower)),
Optional(SECTION_CORE_INTERACTIVE,
default=False): SECTION_CORE_INTERACTIVE_SCHEMA,
# backward compatibility
schema.Optional(SECTION_CORE_CLOUD, default=''): SECTION_CORE_CLOUD_SCHEMA,
schema.Optional(SECTION_CORE_STORAGEPATH, default=''): str,
Optional(SECTION_CORE_CLOUD, default=''): SECTION_CORE_CLOUD_SCHEMA,
Optional(SECTION_CORE_STORAGEPATH, default=''): str,
}
# backward compatibility
@@ -81,10 +114,10 @@ class Config(object):
SECTION_AWS_PROFILE = 'profile'
SECTION_AWS_SCHEMA = {
SECTION_AWS_STORAGEPATH: str,
schema.Optional(SECTION_AWS_REGION): str,
schema.Optional(SECTION_AWS_PROFILE, default='default'): str,
schema.Optional(SECTION_AWS_CREDENTIALPATH, default = ''): str,
schema.Optional(SECTION_AWS_ENDPOINT_URL, default=None): str,
Optional(SECTION_AWS_REGION): str,
Optional(SECTION_AWS_PROFILE, default='default'): str,
Optional(SECTION_AWS_CREDENTIALPATH, default=''): str,
Optional(SECTION_AWS_ENDPOINT_URL, default=None): str,
}
# backward compatibility
@@ -93,7 +126,7 @@ class Config(object):
SECTION_GCP_PROJECTNAME = 'projectname'
SECTION_GCP_SCHEMA = {
SECTION_GCP_STORAGEPATH: str,
schema.Optional(SECTION_GCP_PROJECTNAME): str,
Optional(SECTION_GCP_PROJECTNAME): str,
}
# backward compatibility
@@ -103,31 +136,49 @@ class Config(object):
SECTION_LOCAL_STORAGEPATH: str,
}
SECTION_REMOTE_REGEX = r'^\s*remote\s*"(?P<name>.*)"\s*$'
SECTION_REMOTE_FMT = 'remote "{}"'
SECTION_REMOTE_URL = 'url'
SECTION_REMOTE_USER = 'user'
SECTION_REMOTE_PORT = 'port'
SECTION_REMOTE_KEY_FILE = 'keyfile'
SECTION_REMOTE_TIMEOUT = 'timeout'
SECTION_REMOTE_PASSWORD = 'password'
SECTION_REMOTE_ASK_PASSWORD = 'ask_password'
SECTION_REMOTE_SCHEMA = {
SECTION_REMOTE_URL: supported_url,
schema.Optional(SECTION_AWS_REGION): str,
schema.Optional(SECTION_AWS_PROFILE, default='default'): str,
schema.Optional(SECTION_AWS_CREDENTIALPATH, default = ''): str,
schema.Optional(SECTION_AWS_ENDPOINT_URL, default=None): str,
schema.Optional(SECTION_GCP_PROJECTNAME): str,
schema.Optional(SECTION_CACHE_TYPE): SECTION_CACHE_TYPE_SCHEMA,
schema.Optional(SECTION_REMOTE_USER): str,
SECTION_REMOTE_URL: And(supported_url, error="Unsupported URL"),
Optional(SECTION_AWS_REGION): str,
Optional(SECTION_AWS_PROFILE, default='default'): str,
Optional(SECTION_AWS_CREDENTIALPATH, default=''): str,
Optional(SECTION_AWS_ENDPOINT_URL, default=None): str,
Optional(SECTION_GCP_PROJECTNAME): str,
Optional(SECTION_CACHE_TYPE): SECTION_CACHE_TYPE_SCHEMA,
Optional(SECTION_REMOTE_USER): str,
Optional(SECTION_REMOTE_PORT): Use(int),
Optional(SECTION_REMOTE_KEY_FILE): str,
Optional(SECTION_REMOTE_TIMEOUT): Use(int),
Optional(SECTION_REMOTE_PASSWORD): str,
Optional(SECTION_REMOTE_ASK_PASSWORD): And(str, is_bool, Use(to_bool)),
}
SECTION_STATE = 'state'
SECTION_STATE_ROW_LIMIT = 'row_limit'
SECTION_STATE_ROW_CLEANUP_QUOTA = 'row_cleanup_quota'
SECTION_STATE_SCHEMA = {
Optional(SECTION_STATE_ROW_LIMIT): And(Use(int), is_whole),
Optional(SECTION_STATE_ROW_CLEANUP_QUOTA): And(Use(int), is_percent),
}
SCHEMA = {
schema.Optional(SECTION_CORE, default={}): SECTION_CORE_SCHEMA,
schema.Optional(schema.Regex(SECTION_REMOTE_REGEX)): SECTION_REMOTE_SCHEMA,
schema.Optional(SECTION_CACHE, default={}): SECTION_CACHE_SCHEMA,
Optional(SECTION_CORE, default={}): SECTION_CORE_SCHEMA,
Optional(Regex(SECTION_REMOTE_REGEX)): SECTION_REMOTE_SCHEMA,
Optional(SECTION_CACHE, default={}): SECTION_CACHE_SCHEMA,
Optional(SECTION_STATE, default={}): SECTION_STATE_SCHEMA,
# backward compatibility
schema.Optional(SECTION_AWS, default={}): SECTION_AWS_SCHEMA,
schema.Optional(SECTION_GCP, default={}): SECTION_GCP_SCHEMA,
schema.Optional(SECTION_LOCAL, default={}): SECTION_LOCAL_SCHEMA,
Optional(SECTION_AWS, default={}): SECTION_AWS_SCHEMA,
Optional(SECTION_GCP, default={}): SECTION_GCP_SCHEMA,
Optional(SECTION_LOCAL, default={}): SECTION_LOCAL_SCHEMA,
}
def __init__(self, dvc_dir):
@@ -140,19 +191,31 @@ class Config(object):
local = configobj.ConfigObj(self.config_local_file)
# NOTE: schema doesn't support ConfigObj.Section validation, so we
# need to convert our config to dict before passing it to schema.
# need to convert our config to dict before passing it to
self._config = self._lower(self._config)
local = self._lower(local)
self._config.update(local)
self._config = self._merge(self._config, local)
self._config = schema.Schema(self.SCHEMA).validate(self._config)
self._config = Schema(self.SCHEMA).validate(self._config)
# NOTE: now converting back to ConfigObj
self._config = configobj.ConfigObj(self._config, write_empty_values=True)
self._config = configobj.ConfigObj(self._config,
write_empty_values=True)
self._config.filename = self.config_file
except Exception as ex:
raise ConfigError(ex)
@staticmethod
def _merge(first, second):
res = {}
sections = list(first.keys()) + list(second.keys())
for section in sections:
f = first.get(section, {}).copy()
s = second.get(section, {}).copy()
f.update(s)
res[section] = f
return res
@staticmethod
def _lower(config):
new_config = {}
+278
View File
@@ -0,0 +1,278 @@
import sys
import math
import select
from grandalf.graphs import Vertex, Edge, Graph
from grandalf.layouts import SugiyamaLayout
from grandalf.routing import route_with_lines, EdgeViewer
class AsciiCanvas(object):
def __init__(self, cols, lines):
assert cols > 1
assert lines > 1
self.cols = cols
self.lines = lines
self.canvas = [[' '] * cols for l in range(lines)]
def draw(self):
if sys.stdout.isatty(): # pragma: no cover
from asciimatics.screen import Screen
Screen.wrapper(self._do_draw)
else:
for line in self.canvas:
print(''.join(line))
def _do_draw(self, screen): # pragma: no cover
from asciimatics.event import KeyboardEvent
offset_x = 0
offset_y = 0
smaxrow, smaxcol = screen.dimensions
assert smaxrow > 1
assert smaxcol > 1
smaxrow -= 1
smaxcol -= 1
if self.lines + 1 > smaxrow:
max_y = self.lines + 1 - smaxrow
else:
max_y = 0
if self.cols + 1 > smaxcol:
max_x = self.cols + 1 - smaxcol
else:
max_x = 0
while True:
for y in range(smaxrow + 1):
y_index = offset_y + y
line = []
for x in range(smaxcol + 1):
x_index = offset_x + x
if len(self.canvas) > y_index \
and len(self.canvas[y_index]) > x_index:
line.append(self.canvas[y_index][x_index])
else:
line.append(' ')
assert len(line) == (smaxcol + 1)
screen.print_at(''.join(line), 0, y)
screen.refresh()
# NOTE: get_event() doesn't block by itself,
# so we have to do the blocking ourselves.
select.select([sys.stdin], [], [], None)
event = screen.get_event()
if not isinstance(event, KeyboardEvent):
continue
k = event.key_code
if k == screen.KEY_DOWN or k == ord('s'):
offset_y += 1
elif k == screen.KEY_PAGE_DOWN or k == ord('S'):
offset_y += smaxrow
elif k == screen.KEY_UP or k == ord('w'):
offset_y -= 1
elif k == screen.KEY_PAGE_UP or k == ord('W'):
offset_y -= smaxrow
elif k == screen.KEY_RIGHT or k == ord('d'):
offset_x += 1
elif k == ord('D'):
offset_x += smaxcol
elif k == screen.KEY_LEFT or k == ord('a'):
offset_x -= 1
elif k == ord('A'):
offset_x -= smaxcol
elif k == ord('q') or k == ord('Q'):
break
if offset_y > max_y:
offset_y = max_y
elif offset_y < 0:
offset_y = 0
if offset_x > max_x:
offset_x = max_x
elif offset_x < 0:
offset_x = 0
def point(self, x, y, char):
assert x >= 0
assert x < self.cols
assert y >= 0
assert y < self.lines
self.canvas[y][x] = char
def line(self, x0, y0, x1, y1, char):
if x0 > x1:
x1, x0 = x0, x1
y1, y0 = y0, y1
dx = x1 - x0
dy = y1 - y0
if dx == 0 and dy == 0:
self.point(x0, y0, char)
elif abs(dx) >= abs(dy):
for x in range(x0, x1 + 1):
if dx == 0:
y = y0
else:
y = y0 + int(round((x - x0) * dy / float((dx))))
self.point(x, y, char)
elif y0 < y1:
for y in range(y0, y1 + 1):
if dy == 0:
x = x0
else:
x = x0 + int(round((y - y0) * dx / float((dy))))
self.point(x, y, char)
else:
for y in range(y1, y0 + 1):
if dy == 0:
x = x0
else:
x = x1 + int(round((y - y1) * dx / float((dy))))
self.point(x, y, char)
def text(self, x, y, text):
for i, c in enumerate(text):
self.point(x + i, y, c)
def box(self, x0, y0, w, h):
assert w > 1
assert h > 1
w -= 1
h -= 1
for x in range(x0, x0 + w):
self.point(x, y0, '-')
self.point(x, y0 + h, '-')
for y in range(y0, y0 + h):
self.point(x0, y, '|')
self.point(x0 + w, y, '|')
self.point(x0, y0, '+')
self.point(x0 + w, y0, '+')
self.point(x0, y0 + h, '+')
self.point(x0 + w, y0 + h, '+')
class Dagascii(object):
def __init__(self, vertexes, edges):
#
# Just a reminder about naming conventions:
# +------------X
# |
# |
# |
# |
# Y
#
V = {v: Vertex(" {} ".format(v)) for v in vertexes}
# NOTE: reverting edges to correctly orientate the graph
E = [Edge(V[e], V[s]) for s, e in edges]
V = V.values()
g = Graph(V, E)
class VertexViewer(object):
h = 3 # top and bottom box edges + text
def __init__(self, name):
self.w = len(name) + 2 # right and left bottom edges + text
for v in V:
v.view = VertexViewer(v.data)
# NOTE: determine min box length to create the best layout
minw = min([v.view.w for v in V])
for e in E:
e.view = EdgeViewer()
sug = SugiyamaLayout(g.C[0])
gr = g.C[0]
r = list(filter(lambda x: len(x.e_in()) == 0, gr.sV))
sug.init_all(roots=r, optimize=True)
sug.yspace = VertexViewer.h
sug.xspace = minw
sug.route_edge = route_with_lines
sug.draw()
self.sug = sug
def draw(self):
# NOTE: coordinates might me negative, so we need to shift
# everything to the positive plane before we actually draw it.
Xs = []
Ys = []
for v in self.sug.g.sV:
# NOTE: moving boxes w/2 to the left
Xs.append(v.view.xy[0] - v.view.w/2.0)
Xs.append(v.view.xy[0] + v.view.w/2.0)
Ys.append(v.view.xy[1])
Ys.append(v.view.xy[1] + v.view.h)
for e in self.sug.g.sE:
for xy in e.view._pts:
Xs.append(xy[0])
Ys.append(xy[1])
minx = min(Xs)
miny = min(Ys)
maxx = max(Xs)
maxy = max(Ys)
canvas_cols = int(math.ceil(math.ceil(maxx) - math.floor(minx))) + 1
canvas_lines = int(round(maxy - miny))
canvas = AsciiCanvas(canvas_cols, canvas_lines)
# NOTE: first draw edges so that node boxes could overwrite them
for e in self.sug.g.sE:
assert len(e.view._pts) > 1
for index in range(1, len(e.view._pts)):
start = e.view._pts[index - 1]
end = e.view._pts[index]
start_x = int(round(start[0] - minx))
start_y = int(round(start[1] - miny))
end_x = int(round(end[0] - minx))
end_y = int(round(end[1] - miny))
assert start_x >= 0
assert start_y >= 0
assert end_x >= 0
assert end_y >= 0
canvas.line(start_x, start_y,
end_x, end_y,
'*')
for v in self.sug.g.sV:
# NOTE: moving boxes w/2 to the left
x = v.view.xy[0] - v.view.w/2.0
y = v.view.xy[1]
canvas.box(int(round(x - minx)),
int(round(y - miny)),
v.view.w,
v.view.h)
canvas.text(int(round(x - minx)) + 1,
int(round(y - miny)) + 1,
v.data)
canvas.draw()
+48 -32
View File
@@ -1,38 +1,35 @@
import re
from multiprocessing.pool import ThreadPool
from dvc.logger import Logger
from dvc.exceptions import DvcException
from dvc.config import Config, ConfigError
from dvc.remote.s3 import RemoteS3
from dvc.remote.gs import RemoteGS
from dvc.remote.azure import RemoteAzure
from dvc.remote.ssh import RemoteSSH
from dvc.remote.hdfs import RemoteHDFS
from dvc.remote.local import RemoteLOCAL
from dvc.remote.base import RemoteBase
class DataCloud(object):
""" Generic class to do initial config parsing and redirect to proper DataCloud methods """
CLOUD_MAP = {
'aws' : RemoteS3,
'gcp' : RemoteGS,
'ssh' : RemoteSSH,
'hdfs' : RemoteHDFS,
'local' : RemoteLOCAL,
'aws': RemoteS3,
'gcp': RemoteGS,
'azure': RemoteAzure,
'ssh': RemoteSSH,
'hdfs': RemoteHDFS,
'local': RemoteLOCAL,
}
def __init__(self, project, config=None):
self.project = project
self._config = config
self._core = self._config[Config.SECTION_CORE]
remote = self._config[Config.SECTION_CORE].get(Config.SECTION_CORE_REMOTE, '')
remote = self._core.get(Config.SECTION_CORE_REMOTE, '')
if remote == '':
if config[Config.SECTION_CORE].get(Config.SECTION_CORE_CLOUD, None):
if self._core.get(Config.SECTION_CORE_CLOUD, None):
# backward compatibility
Logger.warn('Using obsoleted config format. Consider updating.')
msg = 'Using obsoleted config format. Consider updating.'
Logger.warn(msg)
self._cloud = self.__init__compat()
else:
self._cloud = None
@@ -51,7 +48,8 @@ class DataCloud(object):
section = Config.SECTION_REMOTE_FMT.format(remote)
cloud_config = self._config.get(section, None)
if not cloud_config:
raise ConfigError("Can't find remote section '{}' in config".format(section))
msg = "Can't find remote section '{}' in config"
raise ConfigError(msg.format(section))
cloud_type = self.supported(cloud_config)
if not cloud_type:
@@ -60,53 +58,71 @@ class DataCloud(object):
return self._init_cloud(cloud_config, cloud_type)
def __init__compat(self):
cloud_name = self._config[Config.SECTION_CORE].get(Config.SECTION_CORE_CLOUD, '').strip().lower()
if cloud_name == '':
name = self._core.get(Config.SECTION_CORE_CLOUD, '').strip().lower()
if name == '':
self._cloud = None
return
cloud_type = self.CLOUD_MAP.get(cloud_name, None)
cloud_type = self.CLOUD_MAP.get(name, None)
if not cloud_type:
raise ConfigError('Wrong cloud type %s specified' % cloud_name)
msg = "Wrong cloud type {} specified"
raise ConfigError(msg.format(name))
cloud_config = self._config.get(cloud_name, None)
cloud_config = self._config.get(name, None)
if not cloud_config:
raise ConfigError('Can\'t find cloud section \'[%s]\' in config' % cloud_name)
msg = "Can't find cloud section '{}' in config"
raise ConfigError(msg.format(name))
return self._init_cloud(cloud_config, cloud_type)
def _init_cloud(self, cloud_config, cloud_type):
global_storage_path = self._config[Config.SECTION_CORE].get(Config.SECTION_CORE_STORAGEPATH, None)
global_storage_path = self._core.get(Config.SECTION_CORE_STORAGEPATH)
if global_storage_path:
Logger.warn('Using obsoleted config format. Consider updating.')
cloud = cloud_type(self.project, cloud_config)
return cloud
def _get_cloud(self, remote):
def _get_cloud(self, remote, cmd):
if remote:
return self._init_remote(remote)
if self._cloud:
return self._cloud
raise ConfigError("No remote repository specified. Setup default repository " \
"with 'dvc config core.remote <name>' or use '-r <name>'.")
msg = "No remote repository specified. Setup default " \
"repository with\n"\
" dvc config core.remote <name>\n" \
"or use:\n" \
" dvc {} -r <name>".format(cmd)
raise ConfigError(msg)
def push(self, targets, jobs=1, remote=None):
def push(self, targets, jobs=1, remote=None, show_checksums=False):
"""
Push data items in a cloud-agnostic way.
"""
return self.project.cache.local.push(targets, jobs=jobs, remote=self._get_cloud(remote))
return self.project.cache.local.push(targets,
jobs=jobs,
remote=self._get_cloud(remote,
'push'),
show_checksums=show_checksums)
def pull(self, targets, jobs=1, remote=None):
def pull(self, targets, jobs=1, remote=None, show_checksums=False):
"""
Pull data items in a cloud-agnostic way.
"""
return self.project.cache.local.pull(targets, jobs=jobs, remote=self._get_cloud(remote))
return self.project.cache.local.pull(targets,
jobs=jobs,
remote=self._get_cloud(remote,
'pull'),
show_checksums=show_checksums)
def status(self, targets, jobs=1, remote=None):
def status(self, targets, jobs=1, remote=None, show_checksums=False):
"""
Check status of data items in a cloud-agnostic way.
"""
return self.project.cache.local.status(targets, jobs=jobs, remote=self._get_cloud(remote))
cloud = self._get_cloud(remote, 'status')
return self.project.cache.local.status(targets,
jobs=jobs,
remote=cloud,
show_checksums=show_checksums)
+8 -5
View File
@@ -18,17 +18,19 @@ from dvc.dependency.ssh import DependencySSH
from dvc.remote import Remote
from dvc.remote.local import RemoteLOCAL
from dvc.remote.s3 import RemoteS3
from dvc.remote.gs import RemoteGS
from dvc.remote.ssh import RemoteSSH
from dvc.remote.hdfs import RemoteHDFS
DEPS = [DependencyHDFS, DependencyS3, DependencyGS, DependencySSH, DependencyLOCAL]
DEPS = [DependencyHDFS,
DependencyS3,
DependencyGS,
DependencySSH,
DependencyLOCAL]
DEP_MAP = {'': DependencyLOCAL,
'ssh': DependencySSH,
's3': DependencyS3,
'gs': DependencyGS,
'hdfs': DependencyHDFS,}
'hdfs': DependencyHDFS}
SCHEMA = {
DependencyBase.PARAM_PATH: str,
@@ -41,7 +43,8 @@ SCHEMA = {
def _get(stage, p, info):
parsed = urlparse(p)
if parsed.scheme == 'remote':
sect = stage.project.config._config[Config.SECTION_REMOTE_FMT.format(parsed.netloc)]
name = Config.SECTION_REMOTE_FMT.format(parsed.netloc)
sect = stage.project.config._config[name]
remote = Remote(stage.project, sect)
return DEP_MAP[remote.scheme](stage, p, info, remote=remote)
+22 -11
View File
@@ -1,24 +1,24 @@
import re
import schema
import posixpath
import ntpath
from dvc.exceptions import DvcException
class DependencyError(DvcException):
def __init__(self, path, msg):
super(DependencyError, self).__init__('Dependency \'{}\' error: {}'.format(path, msg))
msg = 'Dependency \'{}\' error: {}'
super(DependencyError, self).__init__(msg.format(path, msg))
class DependencyDoesNotExistError(DependencyError):
def __init__(self, path):
super(DependencyDoesNotExistError, self).__init__(path, 'does not exist')
msg = 'does not exist'
super(DependencyDoesNotExistError, self).__init__(path, msg)
class DependencyIsNotFileOrDirError(DependencyError):
def __init__(self, path):
super(DependencyIsNotFileOrDirError, self).__init__(path, 'not a file or directory')
msg = 'not a file or directory'
super(DependencyIsNotFileOrDirError, self).__init__(path, msg)
class DependencyBase(object):
@@ -31,6 +31,9 @@ class DependencyBase(object):
self.project = stage.project
self.path = path
def __str__(self):
return self.path
@classmethod
def match(cls, url):
return re.match(cls.REGEX, url)
@@ -43,19 +46,27 @@ class DependencyBase(object):
@classmethod
def supported(cls, url):
return cls.match(url) != None
return cls.match(url) is not None
@property
def sep(self):
return '/'
@property
def exists(self):
return self.remote.exists([self.path_info])[0]
def changed(self):
raise NotImplementedError
def status(self):
if self.changed():
#FIXME better msgs
return {self.rel_path: 'changed'}
# FIXME better msgs
if self.path_info['scheme'] == 'local':
p = self.rel_path
else:
p = self.path
return {p: 'changed'}
return {}
def save(self):
@@ -64,5 +75,5 @@ class DependencyBase(object):
def dumpd(self):
return {self.PARAM_PATH: self.path}
def download(self, path):
self.remote.download([self.path_info], [path])
def download(self, to_info):
self.remote.download([self.path_info], [to_info])
-1
View File
@@ -7,7 +7,6 @@ except ImportError:
from dvc.dependency.s3 import DependencyS3
from dvc.remote.gs import RemoteGS
from dvc.config import Config
class DependencyGS(DependencyS3):
+28 -34
View File
@@ -1,22 +1,19 @@
import os
import schema
import posixpath
import ntpath
try:
from urlparse import urlparse
except ImportError:
from urllib.parse import urlparse
from dvc.dependency.base import *
from dvc.dependency.base import DependencyBase
from dvc.dependency.base import DependencyDoesNotExistError
from dvc.dependency.base import DependencyIsNotFileOrDirError
from dvc.logger import Logger
from dvc.utils import remove
from dvc.config import Config
from dvc.remote.local import RemoteLOCAL
class DependencyLOCAL(DependencyBase):
REGEX = r'^(?P<path>(/+|.:\\+)?.*)$'
REGEX = r'^(?P<path>(/+|.:\\+)?[^:]*)$'
DoesNotExistError = DependencyDoesNotExistError
IsNotFileOrDirError = DependencyIsNotFileOrDirError
@@ -25,19 +22,31 @@ class DependencyLOCAL(DependencyBase):
self.stage = stage
self.project = stage.project
self.info = info
self.remote = remote if remote != None else RemoteLOCAL(stage.project, {})
if remote is not None:
self.remote = remote
else:
self.remote = RemoteLOCAL(stage.project, {})
if remote:
path = os.path.join(remote.prefix, urlparse(path).path.lstrip('/'))
if not os.path.isabs(path):
path = self.ospath(path)
path = self.remote.ospath(path)
path = os.path.join(stage.cwd, path)
self.path = os.path.abspath(os.path.normpath(path))
self.path_info = {'scheme': 'local',
'path': self.path}
def __str__(self):
return self.rel_path
@property
def is_local(self):
assert os.path.isabs(self.path)
assert os.path.isabs(self.project.root_dir)
return self.path.startswith(self.project.root_dir)
@property
def sep(self):
return os.sep
@@ -47,7 +56,7 @@ class DependencyLOCAL(DependencyBase):
return os.path.relpath(self.path)
def changed(self):
if not os.path.exists(self.path):
if not self.exists:
return True
info = self.remote.save_info(self.path_info)
@@ -55,41 +64,26 @@ class DependencyLOCAL(DependencyBase):
return self.info != info
def save(self):
if not os.path.exists(self.path):
if not self.exists:
raise self.DoesNotExistError(self.rel_path)
if not os.path.isfile(self.path) and not os.path.isdir(self.path):
raise self.NotFileOrDirError(self.rel_path)
if not os.path.isfile(self.path) \
and not os.path.isdir(self.path): # pragma: no cover
raise self.IsNotFileOrDirError(self.rel_path)
if (os.path.isfile(self.path) and os.path.getsize(self.path) == 0) or \
(os.path.isdir(self.path) and len(os.listdir(self.path)) == 0):
self.project.logger.warn("File/directory '{}' is empty.".format(self.rel_path))
Logger.warn("File/directory '{}' is empty.".format(self.rel_path))
self.info = self.remote.save_info(self.path_info)
def ospath(self, path):
if os.name == 'nt':
return self.ntpath(path)
return self.unixpath(path)
@staticmethod
def unixpath(path):
assert not ntpath.isabs(path)
assert not posixpath.isabs(path)
return path.replace('\\', '/')
@staticmethod
def ntpath(path):
assert not ntpath.isabs(path)
assert not posixpath.isabs(path)
return path.replace('/', '\\')
def dumpd(self):
if self.path.startswith(self.stage.project.root_dir):
path = self.unixpath(os.path.relpath(self.path, self.stage.cwd))
if self.is_local:
path = self.remote.unixpath(os.path.relpath(self.path,
self.stage.cwd))
else:
path = self.path
info = self.info
info = self.info.copy()
info[self.PARAM_PATH] = path
return info
-1
View File
@@ -7,7 +7,6 @@ except ImportError:
from dvc.dependency.base import DependencyBase
from dvc.remote.s3 import RemoteS3
from dvc.config import Config
class DependencyS3(DependencyBase):
+4 -1
View File
@@ -18,14 +18,17 @@ class DependencySSH(DependencyBase):
self.remote = remote if remote else RemoteSSH(stage.project, {})
host = remote.host if remote else self.group('host')
port = remote.port if remote else RemoteSSH.DEFAULT_PORT
user = remote.user if remote else self.group('user')
if remote:
path = posixpath.join(remote.prefix, urlparse(path).path.lstrip('/'))
path = posixpath.join(remote.prefix,
urlparse(path).path.lstrip('/'))
else:
path = self.match(self.path).group('path')
self.path_info = {'scheme': 'ssh',
'host': host,
'port': port,
'user': user,
'path': path}
+28 -1
View File
@@ -8,6 +8,33 @@ class DvcException(Exception):
if cause:
try:
self.cause_tb = traceback.format_exc()
except Exception:
except Exception: # pragma: no cover
pass
super(DvcException, self).__init__(msg)
class UnsupportedRemoteError(DvcException):
def __init__(self, config):
msg = "Remote '{}' is not supported.".format(config)
super(UnsupportedRemoteError, self).__init__(msg)
class OutputDuplicationError(DvcException):
def __init__(self, output, stages):
assert isinstance(output, str)
assert isinstance(stages, list)
assert all(isinstance(stage, str) for stage in stages)
msg = "File '{}' is specified as an output in more than one stage:{}"
s = ""
for stage in stages:
s += "\n " + stage
super(OutputDuplicationError, self).__init__(msg.format(output, s))
class MoveNotDataSourceError(DvcException):
def __init__(self, path):
msg = "Move is not permitted for stages that are not data sources. " \
"You need to either move '{path}' to a new location and edit " \
"it by hand, or remove '{path}' and create a new one at the " \
"desired location."
super(MoveNotDataSourceError, self).__init__(msg.format(path=path))
+3 -1
View File
@@ -1,4 +1,6 @@
# Based on https://eli.thegreenplace.net/2011/10/19/perls-guess-if-file-is-text-or-binary-implemented-in-python
# Based on https://eli.thegreenplace.net/2011/10/19/
# perls-guess-if-file-is-text-or-binary-implemented-in-python
import sys
PY3 = sys.version_info[0] == 3
+2 -5
View File
@@ -17,15 +17,12 @@ class Lock(object):
self.lock_file = os.path.join(dvc_dir, name)
self._lock = None
@staticmethod
def init(dvc_dir):
return Lock(dvc_dir)
def _do_lock(self):
try:
self._lock = zc.lockfile.LockFile(self.lock_file)
except zc.lockfile.LockError:
raise LockError('Cannot perform the cmd since DVC is busy and locked. Please retry the cmd later.')
raise LockError('Cannot perform the cmd since DVC is busy and '
'locked. Please retry the cmd later.')
def lock(self):
try:
+38 -11
View File
@@ -19,9 +19,16 @@ class Logger(object):
}
COLOR_MAP = {
'debug': colorama.Fore.BLUE,
'warn': colorama.Fore.YELLOW,
'error': colorama.Fore.RED
'green': colorama.Fore.GREEN,
'yellow': colorama.Fore.YELLOW,
'blue': colorama.Fore.BLUE,
'red': colorama.Fore.RED,
}
LEVEL_COLOR_MAP = {
'debug': 'blue',
'warn': 'yellow',
'error': 'red',
}
def __init__(self, loglevel=None):
@@ -58,12 +65,12 @@ class Logger(object):
Logger.logger().setLevel(logging.DEBUG)
@staticmethod
def colorize(msg, typ):
def colorize(msg, color):
header = ''
footer = ''
if sys.stdout.isatty():
header = Logger.COLOR_MAP.get(typ.lower(), '')
if sys.stdout.isatty(): # pragma: no cover
header = Logger.COLOR_MAP.get(color.lower(), '')
footer = colorama.Style.RESET_ALL
return u'{}{}{}'.format(header, msg, footer)
@@ -82,22 +89,42 @@ class Logger(object):
return (str_exc, str_tb)
@staticmethod
def _prefix(msg, typ):
color = Logger.LEVEL_COLOR_MAP.get(typ.lower(), '')
return Logger.colorize('{}: '.format(msg), color)
@staticmethod
def error_prefix():
return Logger._prefix('Error', 'error')
@staticmethod
def warning_prefix():
return Logger._prefix('Warning', 'warn')
@staticmethod
def debug_prefix():
return Logger._prefix('Debug', 'debug')
@staticmethod
def error(msg, exc=None):
prefix = Logger.error_prefix()
str_exc, str_tb = Logger.parse_exc(exc)
if Logger.logger().getEffectiveLevel() == logging.DEBUG and exc:
str_tb = str_tb if str_tb else traceback.format_exc()
Logger.logger().error(str_tb)
return Logger.logger().error(Logger.colorize(msg + str_exc, 'error'))
Logger.logger().error(prefix + str_tb)
chat = "\n\nHaving any troubles? Hit us up at dvc.org/support, " \
"we are always happy to help!"
return Logger.logger().error(prefix + msg + str_exc + chat)
@staticmethod
def warn(msg):
return Logger.logger().warn(Logger.colorize(msg, 'warn'))
return Logger.logger().warn(Logger.warning_prefix() + msg)
@staticmethod
def debug(msg):
return Logger.logger().debug(Logger.colorize(msg, 'debug'))
return Logger.logger().debug(Logger.debug_prefix() + msg)
@staticmethod
def info(msg):
return Logger.logger().info(Logger.colorize(msg, 'info'))
return Logger.logger().info(msg)
+1 -1
View File
@@ -1,6 +1,6 @@
from dvc.logger import Logger
from dvc.cli import parse_args
from dvc.command.common.base import CmdBase
from dvc.command.base import CmdBase
def main(argv=None):
+9 -3
View File
@@ -23,15 +23,21 @@ OUTS_MAP = {'hdfs': OutputHDFS,
'': OutputLOCAL}
SCHEMA[schema.Optional(OutputLOCAL.PARAM_CACHE)] = bool
SCHEMA[schema.Optional(OutputLOCAL.PARAM_METRIC)] = bool
SCHEMA[schema.Optional(OutputLOCAL.PARAM_METRIC)] = OutputLOCAL.METRIC_SCHEMA
def _get(stage, p, info, cache, metric):
parsed = urlparse(p)
if parsed.scheme == 'remote':
sect = stage.project.config._config[Config.SECTION_REMOTE_FMT.format(parsed.netloc)]
name = Config.SECTION_REMOTE_FMT.format(parsed.netloc)
sect = stage.project.config._config[name]
remote = Remote(stage.project, sect)
return OUTS_MAP[remote.scheme](stage, p, info, cache=cache, remote=remote, metric=metric)
return OUTS_MAP[remote.scheme](stage,
p,
info,
cache=cache,
remote=remote,
metric=metric)
for o in OUTS:
if o.supported(p):
+6 -3
View File
@@ -3,7 +3,8 @@ from dvc.exceptions import DvcException
class OutputError(DvcException):
def __init__(self, path, msg):
super(OutputError, self).__init__('Output \'{}\' error: {}'.format(path, msg))
msg = 'Output \'{}\' error: {}'.format(path, msg)
super(OutputError, self).__init__(msg)
class OutputDoesNotExistError(OutputError):
@@ -13,9 +14,11 @@ class OutputDoesNotExistError(OutputError):
class OutputIsNotFileOrDirError(OutputError):
def __init__(self, path):
super(OutputIsNotFileOrDirError, self).__init__(path, 'not a file or directory')
msg = 'not a file or directory'
super(OutputIsNotFileOrDirError, self).__init__(path, msg)
class OutputAlreadyTrackedError(OutputError):
def __init__(self, path):
super(OutputAlreadyTrackedError, self).__init__(path, 'already tracked by scm(e.g. git)')
msg = 'already tracked by scm(e.g. git)'
super(OutputAlreadyTrackedError, self).__init__(path, msg)
+10 -4
View File
@@ -6,11 +6,17 @@ class OutputGS(DependencyGS):
PARAM_CACHE = 'cache'
PARAM_METRIC = 'metric'
def __init__(self, stage, path, info=None, remote=None, cache=True, metric=False):
def __init__(self,
stage,
path,
info=None,
remote=None,
cache=True,
metric=False):
super(OutputGS, self).__init__(stage, path, info, remote=remote)
self.use_cache = cache
self.metric = metric
if cache and self.project.cache.gs == None:
if cache and self.project.cache.gs is None:
raise DvcException("No cache location setup for \'gs\' outputs.")
def dumpd(self):
@@ -23,7 +29,8 @@ class OutputGS(DependencyGS):
if super(OutputGS, self).changed():
return True
if self.use_cache and self.info != self.project.cache.gs.save_info(self.path_info):
if self.use_cache \
and self.project.cache.gs.changed(self.path_info, self.info):
return True
return False
@@ -40,7 +47,6 @@ class OutputGS(DependencyGS):
if not self.use_cache:
return
self.info = self.project.cache.gs.save(self.path_info)
def remove(self, ignore_remove=False):
+10 -3
View File
@@ -6,11 +6,17 @@ class OutputHDFS(DependencyHDFS):
PARAM_CACHE = 'cache'
PARAM_METRIC = 'metric'
def __init__(self, stage, path, info=None, remote=None, cache=True, metric=False):
def __init__(self,
stage,
path,
info=None,
remote=None,
cache=True,
metric=False):
super(OutputHDFS, self).__init__(stage, path, info, remote=remote)
self.use_cache = cache
self.metric = metric
if cache and self.project.cache.hdfs == None:
if cache and self.project.cache.hdfs is None:
raise DvcException("No cache location setup for \'hdfs\' outputs.")
def dumpd(self):
@@ -23,7 +29,8 @@ class OutputHDFS(DependencyHDFS):
if super(OutputHDFS, self).changed():
return True
if self.use_cache and self.info != self.project.cache.hdfs.save_info(self.path_info):
if self.use_cache \
and self.project.cache.hdfs.changed(self.path_info, self.info):
return True
return False
+46 -22
View File
@@ -1,23 +1,33 @@
import os
import schema
import posixpath
import ntpath
from schema import Optional, Or
from dvc.output.base import *
from dvc.output.base import OutputDoesNotExistError, OutputIsNotFileOrDirError
from dvc.output.base import OutputAlreadyTrackedError
from dvc.dependency.local import DependencyLOCAL
from dvc.exceptions import DvcException
from dvc.logger import Logger
from dvc.utils import remove
from dvc.istextfile import istextfile
class OutputLOCAL(DependencyLOCAL):
PARAM_CACHE = 'cache'
PARAM_METRIC = 'metric'
PARAM_METRIC_TYPE = 'type'
PARAM_METRIC_XPATH = 'xpath'
METRIC_SCHEMA = Or(None, bool,
{Optional(PARAM_METRIC_TYPE): Or(str, None),
Optional(PARAM_METRIC_XPATH): Or(str, None)})
DoesNotExistError = OutputDoesNotExistError
IsNotFileOrDirError = OutputIsNotFileOrDirError
def __init__(self, stage, path, info=None, remote=None, cache=True, metric=False):
def __init__(self,
stage,
path,
info=None,
remote=None,
cache=True,
metric=False):
super(OutputLOCAL, self).__init__(stage, path, info, remote=remote)
self.use_cache = cache
self.metric = metric
@@ -30,53 +40,67 @@ class OutputLOCAL(DependencyLOCAL):
def cache(self):
return self.project.cache.local.get(self.md5)
@property
def is_local(self):
return self.path.startswith(self.project.root_dir)
def dumpd(self):
ret = super(OutputLOCAL, self).dumpd()
ret[self.PARAM_CACHE] = self.use_cache
ret[self.PARAM_METRIC] = self.metric
if isinstance(self.metric, dict):
if self.PARAM_METRIC_XPATH in self.metric and \
not self.metric[self.PARAM_METRIC_XPATH]:
del self.metric[self.PARAM_METRIC_XPATH]
if self.metric:
ret[self.PARAM_METRIC] = self.metric
return ret
def changed(self):
if not self.use_cache:
return super(OutputLOCAL, self).changed()
return self.info != self.project.cache.local.save_info(self.path_info)
return self.project.cache.local.changed(self.path_info, self.info)
def checkout(self):
if not self.use_cache:
return
if not self.changed():
msg = u'Data file \'{}\' didn\'t change, skipping checkout.'
self.project.logger.debug(msg.format(self.rel_path))
self.project.cache.local.checkout(self.path_info, self.info)
def _verify_metric(self):
if not self.metric:
return
self.project.cache.local.checkout(self.path_info, self.info)
if os.path.isdir(self.path):
msg = 'Directory \'{}\' cannot be used as metrics.'
raise DvcException(msg.format(self.rel_path))
if not istextfile(self.path):
msg = 'Binary file \'{}\' cannot be used as metrics.'
raise DvcException(msg.format(self.rel_path))
def save(self):
if not self.use_cache:
super(OutputLOCAL, self).save()
self._verify_metric()
msg = 'Output \'{}\' doesn\'t use cache. Skipping saving.'
self.project.logger.debug(msg.format(self.rel_path))
self.project.logger.info(msg.format(self.rel_path))
return
if not os.path.exists(self.path):
raise self.DoesNotExistError(self.rel_path)
if not os.path.isfile(self.path) and not os.path.isdir(self.path):
raise self.NotFileOrDirError(self.rel_path)
if not os.path.isfile(self.path) \
and not os.path.isdir(self.path): # pragma: no cover
raise self.IsNotFileOrDirError(self.rel_path)
if (os.path.isfile(self.path) and os.path.getsize(self.path) == 0) or \
(os.path.isdir(self.path) and len(os.listdir(self.path)) == 0):
self.project.logger.warn("File/directory '{}' is empty.".format(self.rel_path))
msg = "File/directory '{}' is empty.".format(self.rel_path)
self.project.logger.warn(msg)
if not self.changed():
msg = 'Output \'{}\' didn\'t change. Skipping saving.'
self.project.logger.debug(msg.format(self.rel_path))
self.project.logger.info(msg.format(self.rel_path))
return
if self.is_local:
+10 -3
View File
@@ -6,11 +6,17 @@ class OutputS3(DependencyS3):
PARAM_CACHE = 'cache'
PARAM_METRIC = 'metric'
def __init__(self, stage, path, info=None, remote=None, cache=True, metric=False):
def __init__(self,
stage,
path,
info=None,
remote=None,
cache=True,
metric=False):
super(OutputS3, self).__init__(stage, path, info, remote=remote)
self.use_cache = cache
self.metric = metric
if cache and self.project.cache.s3 == None:
if cache and self.project.cache.s3 is None:
raise DvcException("No cache location setup for \'s3\' outputs.")
def dumpd(self):
@@ -23,7 +29,8 @@ class OutputS3(DependencyS3):
if super(OutputS3, self).changed():
return True
if self.use_cache and self.info != self.project.cache.s3.save_info(self.path_info):
if self.use_cache \
and self.project.cache.s3.changed(self.path_info, self.info):
return True
return False
+10 -3
View File
@@ -6,11 +6,17 @@ class OutputSSH(DependencySSH):
PARAM_CACHE = 'cache'
PARAM_METRIC = 'metric'
def __init__(self, stage, path, info=None, remote=None, cache=True, metric=False):
def __init__(self,
stage,
path,
info=None,
remote=None,
cache=True,
metric=False):
super(OutputSSH, self).__init__(stage, path, info, remote=remote)
self.use_cache = cache
self.metric = metric
if cache and self.project.cache.ssh == None:
if cache and self.project.cache.ssh is None:
raise DvcException("No cache location setup for \'ssh\' outputs.")
def dumpd(self):
@@ -23,7 +29,8 @@ class OutputSSH(DependencySSH):
if super(OutputSSH, self).changed():
return True
if self.use_cache and self.info != self.project.cache.ssh.save_info(self.path_info):
if self.use_cache \
and self.project.cache.ssh.changed(self.path_info, self.info):
return True
return False
+12 -7
View File
@@ -1,6 +1,7 @@
from __future__ import print_function
import sys
import threading
import threading
class Progress(object):
"""
@@ -15,6 +16,10 @@ class Progress(object):
self._n_total = total
self._n_finished = 0
@property
def is_finished(self):
return self._n_total == self._n_finished
def _clearln(self):
print('\r\x1b[K', end='')
@@ -52,10 +57,9 @@ class Progress(object):
Make a progress bar out of info, which looks like:
(1/2): [########################################] 100% master.zip
"""
total_len = 100
bar_len = 30
if total == None:
if total is None:
progress = 0
percent = "?% "
else:
@@ -63,15 +67,16 @@ class Progress(object):
progress = int((100 * current)/total) if current < total else 100
percent = str(progress) + "% "
num = "({}/{}): ".format(self._n_finished + 1, self._n_total)
if self._n_total > 1:
num = "({}/{}): ".format(self._n_finished + 1, self._n_total)
else:
num = ""
n_sh = int((progress * bar_len)/100)
n_sp = bar_len - n_sh
bar = "[" + '#'*n_sh + ' '*n_sp + "] "
name_len = total_len - len(num + bar + percent)
name = target_name[:name_len] if len(target_name) > name_len else target_name
return num + bar + percent + target_name
return num + bar + percent + name
progress = Progress()
+644 -200
View File
File diff suppressed because it is too large Load Diff
+40
View File
@@ -0,0 +1,40 @@
import sys
from dvc.progress import progress
try:
# NOTE: in Python3 raw_input() was renamed to input()
input = raw_input
except NameError:
pass
class Prompt(object):
def __init__(self):
self.default = None
def prompt(self, msg, default=False): # pragma: no cover
if self.default is not None:
return self.default
if not sys.stdout.isatty():
return default
answer = input(msg + u' (y/n)\n').lower()
while answer not in ['yes', 'no', 'y', 'n']:
answer = input('Enter \'yes\' or \'no\'.\n').lower()
return answer[0] == "y"
def prompt_password(self, msg): # pragma: no cover
import getpass
if not sys.stdout.isatty():
return None
msg = 'Enter password for {}:\n'.format(msg)
if not progress.is_finished:
msg = u'\n' + msg
return getpass.getpass(msg)
+4 -3
View File
@@ -3,12 +3,13 @@ from dvc.remote.s3 import RemoteS3
from dvc.remote.gs import RemoteGS
from dvc.remote.hdfs import RemoteHDFS
from dvc.remote.ssh import RemoteSSH
from dvc.remote.azure import RemoteAzure
from dvc.config import Config
from dvc.exceptions import DvcException
from dvc.exceptions import UnsupportedRemoteError
REMOTES = [RemoteHDFS, RemoteSSH, RemoteS3, RemoteGS, RemoteLOCAL]
REMOTES = [RemoteHDFS, RemoteSSH, RemoteS3, RemoteGS, RemoteAzure, RemoteLOCAL]
def supported_url(url):
@@ -23,4 +24,4 @@ def Remote(project, config):
for r in REMOTES:
if r.supported(config):
return r(project, config)
raise DvcException('Remote \'{}\' is not supported.'.format(config))
raise UnsupportedRemoteError(str(config))
+244
View File
@@ -0,0 +1,244 @@
from __future__ import absolute_import
import os
import re
try:
from azure.storage.blob import BlockBlobService
except ImportError:
BlockBlobService = None
from dvc.logger import Logger
from dvc.progress import progress
from dvc.config import Config
from dvc.remote.base import RemoteBase
class Callback(object):
def __init__(self, name):
self.name = name
def __call__(self, current, total):
progress.update_target(self.name, current, total)
class RemoteAzure(RemoteBase):
scheme = 'azure'
REGEX = (r'^azure://'
r'(ContainerName=(?P<container_name>[^;]+);?)?'
r'(?P<connection_string>.+)?$')
REQUIRES = {'azure-storage-blob': BlockBlobService}
PARAM_ETAG = 'etag'
COPY_POLL_SECONDS = 5
def __init__(self, project, config):
super(RemoteAzure, self).__init__(project, config)
self.project = project
self.url = config.get(Config.SECTION_REMOTE_URL)
match = re.match(self.REGEX, self.url)
self.bucket = (
match.group('container_name')
or os.getenv('AZURE_STORAGE_CONTAINER_NAME'))
self.connection_string = (
match.group('connection_string')
or os.getenv('AZURE_STORAGE_CONNECTION_STRING'))
if not self.bucket:
raise ValueError('Azure Storage container name missing')
if not self.connection_string:
raise ValueError('Azure Storage connection string missing')
self.__blob_service = None
@property
def blob_service(self):
if self.__blob_service is None:
self.__blob_service = BlockBlobService(
connection_string=self.connection_string)
self.__blob_service.create_container(self.bucket)
return self.__blob_service
# FIXME: temporarily disabled because of the lack of test for external azure
# dependencies/outputs/cache.
#
# def _get_etag(self, bucket, key):
# try:
# blob = self.blob_service.get_blob_properties(bucket, key)
# return blob.properties.etag
# except Exception:
# return None
#
# def save_info(self, path_info):
# if path_info['scheme'] != self.scheme:
# raise NotImplementedError
#
# return {self.PARAM_ETAG: self._get_etag(
# path_info['bucket'], path_info['key'])}
#
# def save(self, path_info):
# if path_info['scheme'] != self.scheme:
# raise NotImplementedError
#
# etag = self._get_etag(path_info['bucket'], path_info['key'])
# dest_key = '{}/{}'.format(etag[0:2], etag[2:])
#
# self._copy_blob(to_bucket=self.bucket,
# to_key=dest_key,
# from_bucket=path_info['bucket'],
# from_key=path_info['key'])
#
# return {self.PARAM_ETAG: etag}
#
# def changed(self, path_info, checksum_info):
# return checksum_info != self.save_info(path_info)
#
# def _copy_blob(self, to_bucket, to_key, from_bucket, from_key):
# source = self.blob_service.make_blob_url(from_bucket, from_key)
#
# copy = self.blob_service.copy_blob(to_bucket, to_key, source)
#
# if self.COPY_POLL_SECONDS <= 0:
# return
#
# while copy.status != 'success':
# time.sleep(self.COPY_POLL_SECONDS)
# copy = self.blob_service.get_blob_properties(
# to_bucket, to_key).properties.copy
#
# def checkout(self, path_info, checksum_info):
# if path_info['scheme'] != self.scheme:
# raise NotImplementedError
#
# etag = checksum_info.get(self.PARAM_ETAG, None)
# if not etag:
# return
#
# self._copy_blob(to_bucket=path_info['bucket'],
# to_key=path_info['key'],
# from_bucket=self.bucket,
# from_key='{}/{}'.format(etag[0:2], etag[2:]))
#
# def remove(self, path_info):
# if path_info['scheme'] != self.scheme:
# raise NotImplementedError
#
# Logger.debug('Removing azure://{}/{}'.format(path_info['bucket'],
# path_info['key']))
#
# self.blob_service.delete_blob(path_info['bucket'], path_info['key'])
def md5s_to_path_infos(self, md5s):
return [{
'scheme': self.scheme,
'bucket': self.bucket,
'key': '{}/{}'.format(md5[0:2], md5[2:])
} for md5 in md5s]
def exists(self, path_infos):
keys = {blob.name
for blob in self.blob_service.list_blobs(self.bucket)}
ret = []
for path_info in path_infos:
if path_info['scheme'] != self.scheme:
raise NotImplementedError
key = path_info['key']
ret.append(key in keys)
return ret
def upload(self, from_infos, to_infos, names=None):
names = self._verify_path_args(to_infos, from_infos, names)
for from_info, to_info, name in zip(from_infos, to_infos, names):
if to_info['scheme'] != self.scheme:
raise NotImplementedError
if from_info['scheme'] != 'local':
raise NotImplementedError
bucket = to_info['bucket']
key = to_info['key']
Logger.debug("Uploading '{}' to '{}/{}'".format(
from_info['path'], bucket, key))
if not name:
name = os.path.basename(from_info['path'])
cb = Callback(name)
try:
self.blob_service.create_blob_from_path(
bucket, key, from_info['path'], progress_callback=cb)
except Exception as ex:
Logger.error("Failed to upload '{}'".format(from_info['path']),
ex)
else:
progress.finish_target(name)
def download(self,
from_infos,
to_infos,
no_progress_bar=False,
names=None):
names = self._verify_path_args(from_infos, to_infos, names)
for to_info, from_info, name in zip(to_infos, from_infos, names):
if from_info['scheme'] != self.scheme:
raise NotImplementedError
if to_info['scheme'] != 'local':
raise NotImplementedError
bucket = from_info['bucket']
key = from_info['key']
Logger.debug("Downloading '{}/{}' to '{}'".format(
bucket, key, to_info['path']))
tmp_file = self.tmp_file(to_info['path'])
if not name:
name = os.path.basename(to_info['path'])
cb = None if no_progress_bar else Callback(name)
self._makedirs(to_info['path'])
try:
self.blob_service.get_blob_to_path(
bucket, key, tmp_file, progress_callback=cb)
except Exception as exc:
Logger.error("Failed to download '{}/{}'".format(
bucket, key), exc)
else:
os.rename(tmp_file, to_info['path'])
if not no_progress_bar:
progress.finish_target(name)
# FIXME: temporarily disabled because of the lack of test for external azure
# dependencies/outputs/cache.
#
# def gc(self, cinfos):
# used = [info[self.PARAM_ETAG] for info in cinfos['azure']]
# used += [info[RemoteLOCAL.PARAM_MD5] for info in cinfos['local']]
#
# all_blobs = self.blob_service.list_blobs(self.bucket)
#
# removed = False
# for blob in all_blobs:
# etag = blob.properties.etag
# if etag in used:
# continue
# path_info = {'scheme': self.scheme,
# 'key': blob.name,
# 'bucket': self.bucket}
# self.remove(path_info)
# removed = True
#
# return removed
+49 -14
View File
@@ -1,7 +1,7 @@
import os
import re
import tempfile
import posixpath
import errno
from multiprocessing import cpu_count
from dvc.config import Config
from dvc.logger import Logger
@@ -15,10 +15,10 @@ STATUS_DELETED = 4
STATUS_MAP = {
# (local_exists, remote_exists)
(True, True) : STATUS_OK,
(False, False) : STATUS_OK,
(True, False) : STATUS_NEW,
(False, True) : STATUS_DELETED,
(True, True): STATUS_OK,
(False, False): STATUS_OK,
(True, False): STATUS_NEW,
(False, True): STATUS_DELETED,
}
@@ -28,10 +28,10 @@ class DataCloudError(DvcException):
super(DataCloudError, self).__init__('Data sync error: {}'.format(msg))
class RemoteBase(object):
REGEX = None
REQUIRES = {}
JOBS = 4 * cpu_count()
def __init__(self, project, config):
pass
@@ -42,9 +42,26 @@ class RemoteBase(object):
url_ok = cls.match(url)
deps_ok = all(cls.REQUIRES.values())
if url_ok and not deps_ok:
missing = [k for k,v in cls.REQUIRES.items() if v == None]
msg = "URL \'{}\' is supported but requires these missing dependencies: {}"
Logger.warn(msg.format(url, str(missing)))
missing = [k for k, v in cls.REQUIRES.items() if v is None]
msg = "URL \'{}\' is supported but requires these missing " \
"dependencies: {}. If you have installed dvc using pip, " \
"choose one of these options to proceed: \n" \
"\n" \
" 1) Install specific missing dependencies:\n" \
" pip install {}\n" \
" 2) Install dvc package that includes those missing " \
"dependencies: \n" \
" pip install dvc[{}]\n" \
" 3) Install dvc package with all possible " \
"dependencies included: \n" \
" pip install dvc[all]\n" \
"\n" \
"If you have installed dvc from a binary package and you " \
"are still seeing this message, please report it to us " \
"using https://github.com/iterative/dvc/issues. Thank you!"
msg = msg.format(url, missing, " ".join(missing), cls.scheme)
Logger.warn(msg)
return url_ok and deps_ok
@classmethod
@@ -60,22 +77,25 @@ class RemoteBase(object):
@staticmethod
def tmp_file(fname):
""" Temporary name for a partial download """
#FIXME probably better use uuid()
# FIXME probably better use uuid()
return fname + '.part'
def save_info(self, path_info):
raise NotImplementedError
def changed(self, path_info, checksum_info):
raise NotImplementedError
def save(self, path_info):
raise NotImplementedError
def checkout(self, path_info, checksum_info):
raise NotImplementedError
def download(self, path_info, path, no_progress_bar=False, name=None):
def download(self, from_infos, to_infos, no_progress_bar=False, name=None):
raise NotImplementedError
def upload(self, path, path_info, name=None):
def upload(self, from_infos, to_infos, path_info, name=None):
raise NotImplementedError
def remove(self, path_info):
@@ -86,10 +106,11 @@ class RemoteBase(object):
def _makedirs(self, fname):
dname = os.path.dirname(fname)
try:
os.makedirs(dname)
except OSError as e:
if e.errno != os.errno.EEXIST:
if e.errno != errno.EEXIST:
raise
def md5s_to_path_infos(self, md5s):
@@ -97,3 +118,17 @@ class RemoteBase(object):
def exists(self, path_infos):
raise NotImplementedError
@classmethod
def _verify_path_args(cls, from_infos, to_infos, names=None):
assert isinstance(from_infos, list)
assert isinstance(to_infos, list)
assert len(from_infos) == len(to_infos)
if not names:
names = len(to_infos) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(to_infos)
return names
+140 -61
View File
@@ -13,8 +13,10 @@ except ImportError:
from dvc.logger import Logger
from dvc.remote.base import RemoteBase
from dvc.remote.local import RemoteLOCAL
from dvc.config import Config
from dvc.progress import progress
from dvc.exceptions import DvcException
class RemoteGS(RemoteBase):
@@ -25,7 +27,9 @@ class RemoteGS(RemoteBase):
def __init__(self, project, config):
self.project = project
storagepath = 'gs://' + config.get(Config.SECTION_AWS_STORAGEPATH, '/').lstrip('/')
storagepath = 'gs://'
storagepath += config.get(Config.SECTION_AWS_STORAGEPATH, '/')
storagepath.lstrip('/')
self.url = config.get(Config.SECTION_REMOTE_URL, storagepath)
self.projectname = config.get(Config.SECTION_GCP_PROJECTNAME, None)
@@ -52,20 +56,62 @@ class RemoteGS(RemoteBase):
if path_info['scheme'] != 'gs':
raise NotImplementedError
return {self.PARAM_ETAG: self.get_etag(path_info['bucket'], path_info['key'])}
return {self.PARAM_ETAG: self.get_etag(path_info['bucket'],
path_info['key'])}
@staticmethod
def to_string(path_info):
return "{}://{}/{}".format(path_info['scheme'],
path_info['bucket'],
path_info['key'])
def changed_cache(self, etag):
key = posixpath.join(self.prefix, etag[0:2], etag[2:])
cache = {'scheme': 'gs', 'bucket': self.bucket, 'key': key}
if {self.PARAM_ETAG: etag} != self.save_info(cache):
if self.exists([cache])[0]:
msg = 'Corrupted cache file {}'
Logger.warn(msg.format(self.to_string(cache)))
self.remove(cache)
return True
return False
def changed(self, path_info, checksum_info):
if not self.exists([path_info])[0]:
return True
etag = checksum_info.get(self.PARAM_ETAG, None)
if etag is None:
return True
if self.changed_cache(etag):
return True
return checksum_info != self.save_info(path_info)
def _copy(self, from_info, to_info, gs=None):
gs = gs if gs else self.gs
blob = gs.bucket(from_info['bucket']).get_blob(from_info['key'])
if not blob:
msg = '{} doesn\'t exist in the cloud'
raise DvcException(msg.format(from_info['key']))
bucket = self.gs.bucket(to_info['bucket'])
bucket.copy_blob(blob,
self.gs.bucket(to_info['bucket']),
new_name=to_info['key'])
def save(self, path_info):
if path_info['scheme'] != 'gs':
raise NotImplementedError
etag = self.get_etag(path_info['bucket'], path_info['key'])
dest_key = posixpath.join(self.prefix, etag[0:2], etag[2:])
key = posixpath.join(self.prefix, etag[0:2], etag[2:])
to_info = {'scheme': 'gs', 'bucket': self.bucket, 'key': key}
blob = self.gs.bucket(path_info['bucket']).get_blob(path_info['key'])
if not blob:
raise DvcException('{} doesn\'t exist in the cloud'.format(path_info['key']))
self.gs.bucket(self.bucket).copy_blob(blob, self.gs.bucket(path_info['bucket']), new_name=dest_key)
self._copy(path_info, to_info)
return {self.PARAM_ETAG: etag}
@@ -77,12 +123,29 @@ class RemoteGS(RemoteBase):
if not etag:
return
key = posixpath.join(self.prefix, etag[0:2], etag[2:])
blob = self.gs.bucket(self.bucket).get_blob(key)
if not blob:
raise DvcException('{} doesn\'t exist in the cloud'.format(key))
if not self.changed(path_info, checksum_info):
msg = "Data '{}' didn't change."
Logger.info(msg.format(self.to_string(path_info)))
return
self.gs.bucket(path_info['bucket']).copy_blob(blob, self.gs.bucket(self.bucket), new_name=path_info['key'])
if self.changed_cache(etag):
msg = "Cache '{}' not found. File '{}' won't be created."
Logger.warn(msg.format(etag, self.to_string(path_info)))
return
if self.exists([path_info])[0]:
msg = "Data '{}' exists. Removing before checkout."
Logger.warn(msg.format(self.to_string(path_info)))
self.remove(path_info)
return
msg = "Checking out '{}' with cache '{}'."
Logger.info(msg.format(self.to_string(path_info), etag))
key = posixpath.join(self.prefix, etag[0:2], etag[2:])
from_info = {'scheme': 'gs', 'bucket': self.bucket, 'key': key}
self._copy(from_info, path_info)
def remove(self, path_info):
if path_info['scheme'] != 'gs':
@@ -100,13 +163,15 @@ class RemoteGS(RemoteBase):
def md5s_to_path_infos(self, md5s):
return [{'scheme': 'gs',
'bucket': self.bucket,
'key': posixpath.join(self.prefix, md5[0:2], md5[2:])} for md5 in md5s]
'key': posixpath.join(self.prefix,
md5[0:2], md5[2:])} for md5 in md5s]
def exists(self, path_infos):
ret = []
gs = self.gs
keys = [blob.name for blob in gs.bucket(self.bucket).list_blobs(prefix=self.prefix)]
bucket = gs.bucket(self.bucket)
keys = [blob.name for blob in bucket.list_blobs(prefix=self.prefix)]
for path_info in path_infos:
exists = False
@@ -115,81 +180,88 @@ class RemoteGS(RemoteBase):
ret.append(exists)
return ret
def upload(self, paths, path_infos, names=None):
assert isinstance(paths, list)
assert isinstance(path_infos, list)
assert len(paths) == len(path_infos)
if not names:
names = len(paths) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(paths)
def upload(self, from_infos, to_infos, names=None):
names = self._verify_path_args(to_infos, from_infos, names)
gs = self.gs
for path, path_info, name in zip(paths, path_infos, names):
if path_info['scheme'] != 'gs':
for from_info, to_info, name in zip(from_infos, to_infos, names):
if to_info['scheme'] != 'gs':
raise NotImplementedError
Logger.debug("Uploading '{}' to '{}/{}'".format(path,
path_info['bucket'],
path_info['key']))
if from_info['scheme'] != 'local':
raise NotImplementedError
Logger.debug("Uploading '{}' to '{}/{}'".format(from_info['path'],
to_info['bucket'],
to_info['key']))
if not name:
name = os.path.basename(path)
name = os.path.basename(from_info['path'])
progress.update_target(name, 0, None)
try:
gs.bucket(path_info['bucket']).blob(path_info['key']).upload_from_filename(path)
bucket = gs.bucket(to_info['bucket'])
blob = bucket.blob(to_info['key'])
blob.upload_from_filename(from_info['path'])
except Exception as exc:
Logger.error("Failed to upload '{}' to '{}/{}'".format(path,
path_info['bucket'],
path_info['key']), exc)
msg = "Failed to upload '{}' to '{}/{}'"
Logger.error(msg.format(from_info['path'],
to_info['bucket'],
to_info['key']), exc)
continue
progress.finish_target(name)
def download(self, path_infos, paths, no_progress_bar=False, names=None):
assert isinstance(paths, list)
assert isinstance(path_infos, list)
assert len(paths) == len(path_infos)
if not names:
names = len(paths) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(paths)
def download(self,
from_infos,
to_infos,
no_progress_bar=False,
names=None):
names = self._verify_path_args(from_infos, to_infos, names)
gs = self.gs
for path, path_info, name in zip(paths, path_infos, names):
if path_info['scheme'] != 'gs':
for to_info, from_info, name in zip(to_infos, from_infos, names):
if from_info['scheme'] != 'gs':
raise NotImplementedError
Logger.debug("Downloading '{}/{}' to '{}'".format(path_info['bucket'],
path_info['key'],
path))
if to_info['scheme'] == 'gs':
self._copy(from_info, to_info, gs=gs)
continue
tmp_file = self.tmp_file(path)
if to_info['scheme'] != 'local':
raise NotImplementedError
msg = "Downloading '{}/{}' to '{}'".format(from_info['bucket'],
from_info['key'],
to_info['path'])
Logger.debug(msg)
tmp_file = self.tmp_file(to_info['path'])
if not name:
name = os.path.basename(path)
name = os.path.basename(to_info['path'])
if not no_progress_bar:
# percent_cb is not available for download_to_filename, so
# lets at least update progress at keypoints(start, finish)
progress.update_target(name, 0, None)
self._makedirs(path)
self._makedirs(to_info['path'])
try:
gs.bucket(path_info['bucket']).get_blob(path_info['key']).download_to_filename(tmp_file)
bucket = gs.bucket(from_info['bucket'])
blob = bucket.get_blob(from_info['key'])
blob.download_to_filename(tmp_file)
except Exception as exc:
Logger.error("Failed to download '{}/{}' to '{}'".format(path_info['bucket'],
path_info['key'],
path), exc)
msg = "Failed to download '{}/{}' to '{}'"
Logger.error(msg.format(from_info['bucket'],
from_info['key'],
to_info['path']), exc)
continue
os.rename(tmp_file, path)
os.rename(tmp_file, to_info['path'])
if not no_progress_bar:
progress.finish_target(name)
@@ -199,16 +271,23 @@ class RemoteGS(RemoteBase):
return posixpath.dirname(relpath) + posixpath.basename(relpath)
def _all_etags(self):
blobs = list(self.gs.bucket(self.bucket).list_blobs(prefix=self.prefix))
blobs = self.gs.bucket(self.bucket).list_blobs(prefix=self.prefix)
blobs = list(blobs)
return [self._path_to_etag(blob.name) for blob in blobs]
def gc(self, checksum_infos):
used_etags = [info[self.PARAM_ETAG] for info in checksum_infos]
def gc(self, cinfos):
used = [info[self.PARAM_ETAG] for info in cinfos['gs']]
used += [info[RemoteLOCAL.PARAM_MD5] for info in cinfos['local']]
removed = False
for etag in self._all_etags():
if etag in used_etags:
if etag in used:
continue
path_info = {'scheme': 'gs',
'bucket': self.bucket,
'key': posixpath.join(self.prefix, etag[0:2], etag[2:])}
'key': posixpath.join(self.prefix,
etag[0:2], etag[2:])}
self.remove(path_info)
removed = True
return removed
+118 -46
View File
@@ -4,19 +4,16 @@ import getpass
import posixpath
from subprocess import Popen, PIPE
try:
from urlparse import urlparse
except ImportError:
from urllib.parse import urlparse
from dvc.config import Config
from dvc.remote.base import RemoteBase
from dvc.remote.local import RemoteLOCAL
from dvc.exceptions import DvcException
from dvc.logger import Logger
from dvc.utils import fix_env
class RemoteHDFS(RemoteBase):
scheme='hdfs'
scheme = 'hdfs'
REGEX = r'^hdfs://((?P<user>.*)@)?.*$'
PARAM_CHECKSUM = 'checksum'
@@ -25,16 +22,23 @@ class RemoteHDFS(RemoteBase):
self.url = config.get(Config.SECTION_REMOTE_URL, '/')
self.user = self.group('user')
if not self.user:
self.user = config.get(Config.SECTION_REMOTE_USER, getpass.getuser())
self.user = config.get(Config.SECTION_REMOTE_USER,
getpass.getuser())
def hadoop_fs(self, cmd, user=None):
cmd = 'hadoop fs -' + cmd
if user:
cmd = 'HADOOP_USER_NAME={} '.format(user) + cmd
# NOTE: close_fds doesn't work with redirected stdin/stdout/stderr.
# See https://github.com/iterative/dvc/issues/1197.
close_fds = (os.name != 'nt')
p = Popen(cmd,
shell=True,
close_fds=True,
close_fds=close_fds,
executable=os.getenv('SHELL'),
env=fix_env(os.environ),
stdin=PIPE,
stdout=PIPE,
stderr=PIPE)
@@ -46,20 +50,24 @@ class RemoteHDFS(RemoteBase):
@staticmethod
def _group(regex, s, gname):
match = re.match(regex, s)
assert match != None
assert match is not None
return match.group(gname)
def checksum(self, path_info):
regex = r'.*\t.*\t(?P<checksum>.*)'
stdout = self.hadoop_fs('checksum {}'.format(path_info['url']), user=path_info['user'])
stdout = self.hadoop_fs('checksum {}'.format(path_info['url']),
user=path_info['user'])
return self._group(regex, stdout, 'checksum')
def cp(self, from_info, to_info):
self.hadoop_fs('mkdir -p {}'.format(posixpath.dirname(to_info['url'])), user=to_info['user'])
self.hadoop_fs('cp {} {}'.format(from_info['url'], to_info['url']), user=to_info['user'])
self.hadoop_fs('mkdir -p {}'.format(posixpath.dirname(to_info['url'])),
user=to_info['user'])
self.hadoop_fs('cp -f {} {}'.format(from_info['url'], to_info['url']),
user=to_info['user'])
def rm(self, path_info):
self.hadoop_fs('rm {}'.format(path_info['url']), user=path_info['user'])
self.hadoop_fs('rm {}'.format(path_info['url']),
user=path_info['user'])
def save_info(self, path_info):
if path_info['scheme'] != 'hdfs':
@@ -69,6 +77,39 @@ class RemoteHDFS(RemoteBase):
return {self.PARAM_CHECKSUM: self.checksum(path_info)}
@staticmethod
def to_string(path_info):
return "{}://{}".format(path_info['scheme'],
path_info['url'])
def changed_cache(self, checksum):
cache = {}
cache['scheme'] = 'hdfs'
cache['user'] = self.user
cache['url'] = posixpath.join(self.url, checksum[0:2], checksum[2:])
if {self.PARAM_CHECKSUM: checksum} != self.save_info(cache):
if self.exists([cache])[0]:
msg = 'Corrupted cache file {}'
Logger.warn(msg.format(self.to_string(cache)))
self.remove(cache)
return True
return False
def changed(self, path_info, checksum_info):
if not self.exists([path_info])[0]:
return True
checksum = checksum_info.get(self.PARAM_CHECKSUM, None)
if checksum is None:
return True
if self.changed_cache(self, checksum):
return True
return checksum_info != self.save_info(path_info)
def save(self, path_info):
if path_info['scheme'] != 'hdfs':
raise NotImplementedError
@@ -93,6 +134,25 @@ class RemoteHDFS(RemoteBase):
if not checksum:
return
if not self.changed(path_info, checksum_info):
msg = "Data '{}' didn't change."
Logger.info(msg.format(self.to_string(path_info)))
return
if self.changed_cache(checksum):
msg = "Cache '{}' not found. File '{}' won't be created."
Logger.warn(msg.format(checksum, self.to_string(path_info)))
return
if self.exists([path_info])[0]:
msg = "Data '{}' exists. Removing before checkout."
Logger.warn(msg.format(self.to_string(path_info)))
self.remove(path_info)
return
msg = "Checking out '{}' with cache '{}'."
Logger.info(msg.format(self.to_string(path_info), checksum))
src = path_info.copy()
src['url'] = posixpath.join(self.url, checksum[0:2], checksum[2:])
@@ -111,7 +171,8 @@ class RemoteHDFS(RemoteBase):
def md5s_to_path_infos(self, md5s):
return [{'scheme': 'hdfs',
'user': self.user,
'url': posixpath.join(self.url, md5[0:2], md5[2:])} for md5 in md5s]
'url': posixpath.join(self.url,
md5[0:2], md5[2:])} for md5 in md5s]
def exists(self, path_infos):
try:
@@ -135,42 +196,47 @@ class RemoteHDFS(RemoteBase):
return ret
def upload(self, paths, path_infos, names=None):
assert isinstance(paths, list)
assert isinstance(path_infos, list)
assert len(paths) == len(path_infos)
if not names:
names = len(paths) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(paths)
def upload(self, from_infos, to_infos, names=None):
names = self._verify_path_args(to_infos, from_infos, names)
for path, path_info, name in zip(paths, path_infos, names):
if path_info['scheme'] != 'hdfs':
for from_info, to_info, name in zip(from_infos, to_infos, names):
if to_info['scheme'] != 'hdfs':
raise NotImplementedError
self.hadoop_fs('mkdir -p {}'.format(posixpath.dirname(path_info['url'])), user=path_info['user'])
self.hadoop_fs('copyFromLocal {} {}'.format(path, path_info['url']), user=path_info['user'])
def download(self, path_infos, paths, no_progress_bar=False, names=None):
assert isinstance(paths, list)
assert isinstance(path_infos, list)
assert len(paths) == len(path_infos)
if not names:
names = len(paths) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(paths)
for path, path_info, name in zip(paths, path_infos, names):
if path_info['scheme'] != 'hdfs':
if from_info['scheme'] != 'local':
raise NotImplementedError
dname = os.path.dirname(path)
cmd = 'mkdir -p {}'.format(posixpath.dirname(to_info['url']))
self.hadoop_fs(cmd, user=to_info['user'])
cmd = 'copyFromLocal {} {}'.format(from_info['path'],
to_info['url'])
self.hadoop_fs(cmd, user=to_info['user'])
def download(self,
from_infos,
to_infos,
no_progress_bar=False,
names=None):
names = self._verify_path_args(from_infos, to_infos, names)
for to_info, from_info, name in zip(to_infos, from_infos, names):
if from_info['scheme'] != 'hdfs':
raise NotImplementedError
if to_info['scheme'] == 'hdfs':
self.cp(from_info, to_info)
continue
if to_info['scheme'] != 'local':
raise NotImplementedError
dname = os.path.dirname(to_info['path'])
if not os.path.exists(dname):
os.makedirs(dname)
self.hadoop_fs('copyToLocal {} {}'.format(path_info['url'], path), user=path_info['user'])
cmd = 'copyToLocal {} {}'.format(from_info['url'], to_info['path'])
self.hadoop_fs(cmd, user=from_info['user'])
def _path_to_checksum(self, path):
relpath = posixpath.relpath(path, self.url)
@@ -186,13 +252,19 @@ class RemoteHDFS(RemoteBase):
flist.append(line.split()[-1])
return [self._path_to_checksum(path) for path in flist]
def gc(self, checksum_infos):
used_checksums = [info[self.PARAM_CHECKSUM] for info in checksum_infos]
def gc(self, cinfos):
used = [info[self.PARAM_CHECKSUM] for info in cinfos['hdfs']]
used += [info[RemoteLOCAL.PARAM_MD5] for info in cinfos['local']]
removed = False
for checksum in self._all_checksums():
if checksum in used_checksums:
if checksum in used:
continue
path_info = {'scheme': 'hdfs',
'user': self.user,
'url': posixpath.join(self.url, checksum[0:2], checksum[2:])}
'url': posixpath.join(self.url,
checksum[0:2], checksum[2:])}
self.remove(path_info)
removed = True
return removed
+364 -110
View File
@@ -1,25 +1,29 @@
import os
import uuid
import json
import ntpath
import shutil
import filecmp
import posixpath
from operator import itemgetter
from dvc.system import System
from dvc.remote.base import RemoteBase, STATUS_MAP
from dvc.state import State
from dvc.logger import Logger
from dvc.utils import remove, move, copyfile, file_md5, to_chunks
from dvc.utils import remove, move, copyfile, dict_md5, to_chunks
from dvc.utils import LARGE_DIR_SIZE
from dvc.config import Config
from dvc.exceptions import DvcException
from dvc.progress import progress
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
from concurrent.futures import ThreadPoolExecutor
class RemoteLOCAL(RemoteBase):
scheme = ''
REGEX = r'^(?P<path>(/+|.:\\+).*)$'
PARAM_MD5 = State.PARAM_MD5
PARAM_RELPATH = State.PARAM_RELPATH
PARAM_MD5 = 'md5'
PARAM_PATH = 'path'
PARAM_RELPATH = 'relpath'
MD5_DIR_SUFFIX = '.dir'
CACHE_TYPES = ['reflink', 'hardlink', 'symlink', 'copy']
CACHE_TYPE_MAP = {
@@ -32,7 +36,6 @@ class RemoteLOCAL(RemoteBase):
def __init__(self, project, config):
self.project = project
self.state = self.project.state
self.link_state = project.link_state
storagepath = config.get(Config.SECTION_AWS_STORAGEPATH, None)
self.cache_dir = config.get(Config.SECTION_REMOTE_URL, storagepath)
@@ -44,9 +47,13 @@ class RemoteLOCAL(RemoteBase):
else:
self.cache_types = self.CACHE_TYPES
if self.cache_dir != None and not os.path.exists(self.cache_dir):
if self.cache_dir is not None and not os.path.exists(self.cache_dir):
os.mkdir(self.cache_dir)
@property
def url(self):
return self.cache_dir
@property
def prefix(self):
return self.cache_dir
@@ -74,11 +81,12 @@ class RemoteLOCAL(RemoteBase):
relpath = os.path.relpath(path, self.cache_dir)
return os.path.dirname(relpath) + os.path.basename(relpath)
def changed(self, md5):
def changed_cache(self, md5):
cache = self.get(md5)
if self.state.changed(cache, md5=md5):
if os.path.exists(cache):
Logger.warn('Corrupted cache file {}'.format(os.path.relpath(cache)))
msg = 'Corrupted cache file {}'
Logger.warn(msg.format(os.path.relpath(cache)))
remove(cache)
return True
@@ -95,15 +103,76 @@ class RemoteLOCAL(RemoteBase):
while i > 0:
try:
self.CACHE_TYPE_MAP[self.cache_types[0]](cache, path)
msg = "Created '{}': {} -> {}"
Logger.info(msg.format(self.cache_types[0],
os.path.relpath(cache),
os.path.relpath(path)))
return
except Exception as exc:
msg = 'Cache type \'{}\' is not supported'.format(self.cache_types[0])
Logger.debug(msg)
except DvcException as exc:
msg = 'Cache type \'{}\' is not supported: {}'
Logger.debug(msg.format(self.cache_types[0], str(exc)))
del self.cache_types[0]
i -= 1
raise DvcException('No possible cache types left to try out.')
@classmethod
def ospath(cls, path):
if os.name == 'nt':
return cls.ntpath(path)
return cls.unixpath(path)
@staticmethod
def unixpath(path):
assert not ntpath.isabs(path)
assert not posixpath.isabs(path)
return path.replace('\\', '/')
@staticmethod
def ntpath(path):
assert not ntpath.isabs(path)
assert not posixpath.isabs(path)
return path.replace('/', '\\')
def collect_dir_cache(self, dname):
dir_info = []
bar = False
for root, dirs, files in os.walk(dname):
if len(files) > LARGE_DIR_SIZE:
msg = "Computing md5 for a large directory {}. " \
"This is only done once."
Logger.info(msg.format(os.path.relpath(dname)))
bar = True
title = os.path.relpath(dname)
processed = 0
total = len(files)
progress.update_target(title, 0, total)
for fname in files:
path = os.path.join(root, fname)
relpath = self.unixpath(os.path.relpath(path, dname))
if bar:
progress.update_target(title, processed, total)
processed += 1
md5 = self.state.update(path)
dir_info.append({self.PARAM_RELPATH: relpath,
self.PARAM_MD5: md5})
if bar:
progress.finish_target(title)
# NOTE: sorting the list by path to ensure reproducibility
dir_info = sorted(dir_info, key=itemgetter(self.PARAM_RELPATH))
md5 = dict_md5(dir_info) + self.MD5_DIR_SUFFIX
if self.changed_cache(md5):
self.dump_dir_cache(md5, dir_info)
return (md5, dir_info)
def load_dir_cache(self, md5):
path = self.get(md5)
@@ -122,6 +191,9 @@ class RemoteLOCAL(RemoteBase):
Logger.error(msg.format(os.path.relpath(path)))
return []
for info in d:
info['relpath'] = self.ospath(info['relpath'])
return d
def dump_dir_cache(self, md5, dir_info):
@@ -141,9 +213,9 @@ class RemoteLOCAL(RemoteBase):
json.dump(dir_info, fd, sort_keys=True)
move(tmp, path)
@staticmethod
def is_dir_cache(cache):
return cache.endswith(State.MD5_DIR_SUFFIX)
@classmethod
def is_dir_cache(cls, cache):
return cache.endswith(cls.MD5_DIR_SUFFIX)
def checkout(self, path_info, checksum_info):
path = path_info['path']
@@ -151,26 +223,32 @@ class RemoteLOCAL(RemoteBase):
cache = self.get(md5)
if not cache:
Logger.warn('No cache info for \'{}\'. Skipping checkout.'.format(os.path.relpath(path)))
msg = 'No cache info for \'{}\'. Skipping checkout.'
Logger.warn(msg.format(os.path.relpath(path)))
return
if self.changed(md5):
if not self.changed(path_info, checksum_info):
msg = "Data '{}' didn't change."
Logger.info(msg.format(os.path.relpath(path)))
return
if self.changed_cache(md5):
msg = u'Cache \'{}\' not found. File \'{}\' won\'t be created.'
Logger.warn(msg.format(md5, os.path.relpath(path)))
remove(path)
return
if os.path.exists(path):
msg = u'Data \'{}\' exists. Removing before checkout'
Logger.debug(msg.format(os.path.relpath(path)))
msg = u'Data \'{}\' exists. Removing before checkout.'
Logger.warn(msg.format(os.path.relpath(path)))
remove(path)
msg = u'Checking out \'{}\' with cache \'{}\''
Logger.debug(msg.format(os.path.relpath(path), md5))
msg = u'Checking out \'{}\' with cache \'{}\'.'
Logger.info(msg.format(os.path.relpath(path), md5))
if not self.is_dir_cache(cache):
self.link(cache, path)
self.link_state.update(path)
self.state.update_link(path)
return
# Create dir separately so that dir is created
@@ -184,7 +262,7 @@ class RemoteLOCAL(RemoteBase):
relpath = entry[self.PARAM_RELPATH]
p = os.path.join(path, relpath)
self.link(c, p)
self.link_state.update(path)
self.state.update_link(path)
def _move(self, inp, outp):
# moving in two stages to make last the move atomic in
@@ -196,17 +274,17 @@ class RemoteLOCAL(RemoteBase):
def _save_file(self, path_info):
path = path_info['path']
md5 = self.state.update(path)
assert md5 != None
assert md5 is not None
cache = self.get(md5)
if self.changed(md5):
if self.changed_cache(md5):
self._move(path, cache)
else:
remove(path)
self.link(cache, path)
self.link_state.update(path)
self.state.update_link(path)
return {self.PARAM_MD5: md5}
@@ -215,19 +293,19 @@ class RemoteLOCAL(RemoteBase):
md5, dir_info = self.state.update_info(path)
for entry in dir_info:
relpath = entry[State.PARAM_RELPATH]
m = entry[State.PARAM_MD5]
relpath = entry[self.PARAM_RELPATH]
m = entry[self.PARAM_MD5]
p = os.path.join(path, relpath)
c = self.get(m)
if self.changed(m):
if self.changed_cache(m):
self._move(p, c)
else:
remove(p)
self.link(c, p)
self.link_state.update(path)
self.state.update_link(path)
return {self.PARAM_MD5: md5}
@@ -237,6 +315,10 @@ class RemoteLOCAL(RemoteBase):
path = path_info['path']
msg = "Saving '{}' to cache '{}'."
Logger.info(msg.format(os.path.relpath(path),
os.path.relpath(self.cache_dir)))
if os.path.isdir(path):
return self._save_dir(path_info)
else:
@@ -248,6 +330,19 @@ class RemoteLOCAL(RemoteBase):
return {self.PARAM_MD5: self.state.update(path_info['path'])}
def changed(self, path_info, checksum_info):
if not self.exists([path_info])[0]:
return True
md5 = checksum_info.get(self.PARAM_MD5, None)
if md5 is None:
return True
if self.changed_cache(md5):
return True
return checksum_info != self.save_info(path_info)
def remove(self, path_info):
if path_info['scheme'] != 'local':
raise NotImplementedError
@@ -262,7 +357,8 @@ class RemoteLOCAL(RemoteBase):
def md5s_to_path_infos(self, md5s):
return [{'scheme': 'local',
'path': os.path.join(self.prefix, md5[0:2], md5[2:])} for md5 in md5s]
'path': os.path.join(self.prefix,
md5[0:2], md5[2:])} for md5 in md5s]
def exists(self, path_infos):
ret = []
@@ -270,153 +366,311 @@ class RemoteLOCAL(RemoteBase):
ret.append(os.path.exists(path_info['path']))
return ret
def upload(self, paths, path_infos, names=None):
assert isinstance(paths, list)
assert isinstance(path_infos, list)
assert len(paths) == len(path_infos)
if not names:
names = len(paths) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(paths)
def upload(self, from_infos, to_infos, names=None):
names = self._verify_path_args(to_infos, from_infos, names)
for path, path_info, name in zip(paths, path_infos, names):
if path_info['scheme'] != 'local':
for from_info, to_info, name in zip(from_infos, to_infos, names):
if to_info['scheme'] != 'local':
raise NotImplementedError
Logger.debug("Uploading '{}' to '{}'".format(path, path_info['path']))
if not name:
name = os.path.basename(path)
self._makedirs(path_info['path'])
try:
copyfile(path, path_info['path'], name=name)
except Exception as exc:
Logger.error("Failed to upload '{}' tp '{}'".format(path, path_info['path']))
def download(self, path_infos, paths, no_progress_bar=False, names=None):
assert isinstance(paths, list)
assert isinstance(path_infos, list)
assert len(paths) == len(path_infos)
if not names:
names = len(paths) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(paths)
for path, path_info, name in zip(paths, path_infos, names):
if path_info['scheme'] != 'local':
if from_info['scheme'] != 'local':
raise NotImplementedError
Logger.debug("Downloading '{}' to '{}'".format(path_info['path'], path))
Logger.debug("Uploading '{}' to '{}'".format(from_info['path'],
to_info['path']))
if not name:
name = os.path.basename(path)
name = os.path.basename(from_info['path'])
self._makedirs(to_info['path'])
self._makedirs(path)
tmp_file = self.tmp_file(path)
try:
copyfile(path_info['path'], tmp_file, no_progress_bar=no_progress_bar, name=name)
copyfile(from_info['path'], to_info['path'], name=name)
except Exception as exc:
Logger.error("Failed to download '{}' to '{}'".format(path_info['path'], path), exc)
msg = "Failed to upload '{}' tp '{}'"
Logger.error(msg.format(from_info['path'],
to_info['path']), exc)
def download(self,
from_infos,
to_infos,
no_progress_bar=False,
names=None):
names = self._verify_path_args(from_infos, to_infos, names)
for to_info, from_info, name in zip(to_infos, from_infos, names):
if from_info['scheme'] != 'local':
raise NotImplementedError
if to_info['scheme'] != 'local':
raise NotImplementedError
Logger.debug("Downloading '{}' to '{}'".format(from_info['path'],
to_info['path']))
if not name:
name = os.path.basename(to_info['path'])
self._makedirs(to_info['path'])
tmp_file = self.tmp_file(to_info['path'])
try:
copyfile(from_info['path'],
tmp_file,
no_progress_bar=no_progress_bar,
name=name)
except Exception as exc:
msg = "Failed to download '{}' to '{}'"
Logger.error(msg.format(from_info['path'],
to_info['path']), exc)
continue
os.rename(tmp_file, path)
os.rename(tmp_file, to_info['path'])
def _collect(self, checksum_infos):
def _collect(self, checksum_infos, push=False):
missing = []
collected = []
for info in checksum_infos:
md5 = info[self.PARAM_MD5]
cache = self.get(md5)
if not self.is_dir_cache(info[self.PARAM_MD5]):
continue
if not os.path.exists(cache):
missing.append(info)
collected.extend(self.load_dir_cache(md5))
continue
for i in self.load_dir_cache(md5):
if info.get('branch'):
i['branch'] = info['branch']
i[self.PARAM_PATH] = os.path.join(info[self.PARAM_PATH],
i[self.PARAM_RELPATH])
collected.append(i.copy())
collected.extend(checksum_infos)
return collected, missing
def gc(self, checksum_infos):
used_md5s = [info[self.PARAM_MD5] for info in self._collect(checksum_infos)[0]]
def _group(self, checksum_infos, show_checksums=False):
by_md5 = {}
for info in checksum_infos:
md5 = info[self.PARAM_MD5]
if show_checksums:
by_md5[md5] = md5
continue
name = info[self.PARAM_PATH]
branch = info.get('branch')
if branch:
name += '({})'.format(branch)
if md5 not in by_md5.keys():
by_md5[md5] = ''
else:
by_md5[md5] += ' '
by_md5[md5] += name
return list(by_md5.keys()), list(by_md5.values())
def gc(self, checksum_infos):
checksum_infos = self._collect(checksum_infos['local'])[0]
used_md5s = [info[self.PARAM_MD5] for info in checksum_infos]
removed = False
for md5 in self.all():
if md5 in used_md5s:
continue
remove(self.get(md5))
removed = True
return removed
def status(self, checksum_infos, remote, jobs=None, show_checksums=False):
Logger.info("Preparing to pull data from {}".format(remote.url))
title = "Collecting information"
progress.set_n_total(1)
progress.update_target(title, 0, 100)
checksum_infos, missing = self._collect(checksum_infos)
checksum_infos += missing
progress.update_target(title, 10, 100)
md5s, names = self._group(checksum_infos,
show_checksums=show_checksums)
progress.update_target(title, 20, 100)
def status(self, checksum_infos, remote, jobs=1):
checksum_infos = self._collect(checksum_infos)[0]
md5s = [info[self.PARAM_MD5] for info in checksum_infos]
path_infos = remote.md5s_to_path_infos(md5s)
progress.update_target(title, 30, 100)
remote_exists = remote.exists(path_infos)
local_exists = [not self.changed(md5) for md5 in md5s]
return [(md5, STATUS_MAP[l,r]) for md5, l, r in zip(md5s, local_exists, remote_exists)]
progress.update_target(title, 90, 100)
def _do_pull(self, checksum_infos, remote, jobs=1, no_progress_bar=False):
md5s = [info[self.PARAM_MD5] for info in checksum_infos]
local_exists = [not self.changed_cache(md5) for md5 in md5s]
progress.finish_target(title)
return [(name, STATUS_MAP[l, r]) for name, l, r in zip(names,
local_exists,
remote_exists)]
def _do_pull(self,
checksum_infos,
remote,
jobs=None,
show_checksums=False):
title = "Collecting information"
progress.set_n_total(1)
progress.update_target(title, 0, 100)
grouped = zip(*self._group(checksum_infos,
show_checksums=show_checksums))
progress.update_target(title, 10, 100)
md5s = []
names = []
# NOTE: filter files that are not corrupted
md5s = list(filter(lambda md5: self.changed(md5), md5s))
for md5, name in grouped:
if self.changed_cache(md5):
md5s.append(md5)
names.append(name)
progress.update_target(title, 30, 100)
cache = [{'scheme': 'local', 'path': self.get(md5)} for md5 in md5s]
progress.update_target(title, 50, 100)
cache = [self.get(md5) for md5 in md5s]
path_infos = remote.md5s_to_path_infos(md5s)
assert len(path_infos) == len(cache) == len(md5s)
progress.update_target(title, 60, 100)
# NOTE: dummy call to try to establish a connection
# to see if we need to ask user for a password.
remote.exists(remote.md5s_to_path_infos(['000']))
progress.update_target(title, 70, 100)
assert len(path_infos) == len(cache) == len(md5s) == len(names)
if jobs is None:
jobs = remote.JOBS
chunks = list(zip(to_chunks(path_infos, jobs),
to_chunks(cache, jobs),
to_chunks(md5s, jobs)))
to_chunks(names, jobs)))
progress.set_n_total(len(md5s))
progress.finish_target(title)
progress.set_n_total(len(names))
if len(chunks) == 0:
return
futures = []
with ThreadPoolExecutor(max_workers=len(chunks)) as executor:
for path_infos, paths, md5s in chunks:
executor.submit(remote.download, path_infos,
paths,
names=md5s,
no_progress_bar=no_progress_bar)
for from_infos, to_infos, names in chunks:
res = executor.submit(remote.download,
from_infos,
to_infos,
names=names)
futures.append(res)
for f in futures:
f.result()
def pull(self, checksum_infos, remote, jobs=None, show_checksums=False):
Logger.info("Preparing to pull data from {}".format(remote.url))
def pull(self, checksum_infos, remote, jobs=1):
# NOTE: try fetching missing dir info
checksum_infos, missing = self._collect(checksum_infos)
if len(missing) > 0:
self._do_pull(missing, remote, jobs, no_progress_bar=True)
self._do_pull(missing,
remote,
jobs,
show_checksums=show_checksums)
checksum_infos += self._collect(missing)[0]
self._do_pull(checksum_infos, remote, jobs)
self._do_pull(checksum_infos,
remote,
jobs,
show_checksums=show_checksums)
def push(self, checksum_infos, remote, jobs=1):
md5s = [info[self.PARAM_MD5] for info in self._collect(checksum_infos)[0]]
def push(self, checksum_infos, remote, jobs=None, show_checksums=False):
Logger.info("Preparing to push data to {}".format(remote.url))
title = "Collecting information"
progress.set_n_total(1)
progress.update_target(title, 0, 100)
checksum_infos = self._collect(checksum_infos)[0]
progress.update_target(title, 10, 100)
# NOTE: verifying that our cache is not corrupted
md5s = list(filter(lambda md5: not self.changed(md5), md5s))
def func(info):
return not self.changed_cache(info[self.PARAM_MD5])
checksum_infos = list(filter(func, checksum_infos))
progress.update_target(title, 20, 100)
# NOTE: filter files that are already uploaded
path_infos = remote.md5s_to_path_infos(md5s)
md5s_exist = filter(lambda x: not x[1], list(zip(md5s, remote.exists(path_infos))))
md5s = [md5 for md5, exists in md5s_exist]
md5s = [i[self.PARAM_MD5] for i in checksum_infos]
exists = remote.exists(remote.md5s_to_path_infos(md5s))
progress.update_target(title, 30, 100)
def func(entry):
return not entry[0]
assert len(exists) == len(checksum_infos)
infos_exist = list(filter(func, zip(exists, checksum_infos)))
checksum_infos = [i for e, i in infos_exist]
progress.update_target(title, 70, 100)
md5s, names = self._group(checksum_infos,
show_checksums=show_checksums)
cache = [{'scheme': 'local', 'path': self.get(md5)} for md5 in md5s]
progress.update_target(title, 80, 100)
cache = [self.get(md5) for md5 in md5s]
path_infos = remote.md5s_to_path_infos(md5s)
assert len(path_infos) == len(cache) == len(md5s)
assert len(path_infos) == len(cache) == len(md5s) == len(names)
progress.update_target(title, 90, 100)
if jobs is None:
jobs = remote.JOBS
chunks = list(zip(to_chunks(path_infos, jobs),
to_chunks(cache, jobs),
to_chunks(md5s, jobs)))
to_chunks(names, jobs)))
progress.set_n_total(len(md5s))
progress.finish_target(title)
progress.set_n_total(len(names))
if len(chunks) == 0:
return
futures = []
with ThreadPoolExecutor(max_workers=len(chunks)) as executor:
for path_infos, paths, md5s in chunks:
executor.submit(remote.upload, paths, path_infos, names=md5s)
for to_infos, from_infos, names in chunks:
res = executor.submit(remote.upload,
from_infos,
to_infos,
names=names)
futures.append(res)
for f in futures:
f.result()
+183 -76
View File
@@ -1,5 +1,4 @@
import os
import math
import threading
import posixpath
@@ -17,6 +16,8 @@ from dvc.logger import Logger
from dvc.progress import progress
from dvc.config import Config
from dvc.remote.base import RemoteBase
from dvc.remote.local import RemoteLOCAL
from dvc.exceptions import DvcException
class Callback(object):
@@ -39,14 +40,30 @@ class RemoteS3(RemoteBase):
PARAM_ETAG = 'etag'
def __init__(self, project, config):
import configobj
self.project = project
storagepath = 's3://' + config.get(Config.SECTION_AWS_STORAGEPATH, '').lstrip('/')
storagepath = 's3://' + config.get(Config.SECTION_AWS_STORAGEPATH,
'').lstrip('/')
self.url = config.get(Config.SECTION_REMOTE_URL, storagepath)
self.region = config.get(Config.SECTION_AWS_REGION, None)
self.profile = config.get(Config.SECTION_AWS_PROFILE, None)
self.credentialpath = config.get(Config.SECTION_AWS_CREDENTIALPATH, None)
self.profile = os.getenv('AWS_PROFILE',
config.get(Config.SECTION_AWS_PROFILE,
'default'))
self.endpoint_url = config.get(Config.SECTION_AWS_ENDPOINT_URL, None)
credentialpath = config.get(Config.SECTION_AWS_CREDENTIALPATH, None)
if credentialpath:
creds_conf = configobj.ConfigObj(credentialpath)
creds = creds_conf.get(self.profile, {})
else:
creds = self._get_aws_config_from_env()
self.creds = creds
self.region = creds.get('region', self.region)
self.aws_access_key_id = creds.get('aws_access_key_id', None)
self.aws_secret_access_key = creds.get('aws_secret_access_key', None)
@property
def bucket(self):
return urlparse(self.url).netloc
@@ -57,13 +74,28 @@ class RemoteS3(RemoteBase):
@property
def s3(self):
return boto3.resource('s3', endpoint_url=self.endpoint_url)
if not self.creds:
session = boto3.session.Session(profile_name=self.profile)
else:
session = boto3.session.Session(
aws_access_key_id=self.aws_access_key_id,
aws_secret_access_key=self.aws_secret_access_key,
region_name=self.region)
return session.client('s3', endpoint_url=self.endpoint_url)
def _get_aws_config_from_env(self):
creds = {
'region': os.getenv('AWS_DEFAULT_REGION'),
'aws_access_key_id': os.getenv('AWS_ACCESS_KEY_ID'),
'aws_secret_access_key': os.getenv('AWS_SECRET_ACCESS_KEY'),
}
return {k: v for k, v in creds.items() if v is not None}
def get_etag(self, bucket, key):
try:
obj = self.s3.Object(bucket, key).get()
obj = self.s3.head_object(Bucket=bucket, Key=key)
except Exception:
return None
raise DvcException('s3://{}/{} does not exist'.format(bucket, key))
return obj['ETag'].strip('"')
@@ -71,21 +103,58 @@ class RemoteS3(RemoteBase):
if path_info['scheme'] != 's3':
raise NotImplementedError
return {self.PARAM_ETAG: self.get_etag(path_info['bucket'], path_info['key'])}
return {self.PARAM_ETAG: self.get_etag(path_info['bucket'],
path_info['key'])}
def changed(self, path_info, checksum_info):
if not self.exists([path_info])[0]:
return True
etag = checksum_info.get(self.PARAM_ETAG, None)
if etag is None:
return True
if self.changed_cache(etag):
return True
return checksum_info != self.save_info(path_info)
def _copy(self, from_info, to_info, s3=None):
s3 = s3 if s3 else self.s3
source = {'Bucket': from_info['bucket'],
'Key': from_info['key']}
self.s3.copy(source, to_info['bucket'], to_info['key'])
def save(self, path_info):
if path_info['scheme'] != 's3':
raise NotImplementedError
etag = self.get_etag(path_info['bucket'], path_info['key'])
dest_key = posixpath.join(self.prefix, etag[0:2], etag[2:])
key = posixpath.join(self.prefix, etag[0:2], etag[2:])
to_info = {'scheme': 's3', 'bucket': self.bucket, 'key': key}
source = {'Bucket': path_info['bucket'],
'Key': path_info['key']}
self.s3.Bucket(self.bucket).copy(source, dest_key)
self._copy(path_info, to_info)
return {self.PARAM_ETAG: etag}
@staticmethod
def to_string(path_info):
return "s3://{}/{}".format(path_info['bucket'], path_info['key'])
def changed_cache(self, etag):
key = posixpath.join(self.prefix, etag[0:2], etag[2:])
cache = {'scheme': 's3', 'bucket': self.bucket, 'key': key}
if {self.PARAM_ETAG: etag} != self.save_info(cache):
if self.exists([cache])[0]:
msg = 'Corrupted cache file {}'
Logger.warn(msg.format(self.to_string(cache)))
self.remove(cache)
return True
return False
def checkout(self, path_info, checksum_info):
if path_info['scheme'] != 's3':
raise NotImplementedError
@@ -94,11 +163,29 @@ class RemoteS3(RemoteBase):
if not etag:
return
key = posixpath.join(self.prefix, etag[0:2], etag[2:])
source = {'Bucket': self.bucket,
'Key': key}
if not self.changed(path_info, checksum_info):
msg = "Data '{}' didn't change."
Logger.info(msg.format(self.to_string(path_info)))
return
self.s3.Bucket(path_info['bucket']).copy(source, path_info['key'])
if self.changed_cache(etag):
msg = "Cache '{}' not found. File '{}' won't be created."
Logger.warn(msg.format(etag, self.to_string(path_info)))
return
if self.exists([path_info])[0]:
msg = "Data '{}' exists. Removing before checkout."
Logger.warn(msg.format(self.to_string(path_info)))
self.remove(path_info)
return
msg = "Checking out '{}' with cache '{}'."
Logger.info(msg.format(self.to_string(path_info), etag))
key = posixpath.join(self.prefix, etag[0:2], etag[2:])
from_info = {'scheme': 's3', 'bucket': self.bucket, 'key': key}
self._copy(from_info, path_info)
def remove(self, path_info):
if path_info['scheme'] != 's3':
@@ -107,21 +194,17 @@ class RemoteS3(RemoteBase):
Logger.debug('Removing s3://{}/{}'.format(path_info['bucket'],
path_info['key']))
try:
obj = self.s3.Object(path_info['bucket'], path_info['key']).get()
obj.delete()
except Exception:
pass
self.s3.delete_object(Bucket=path_info['bucket'],
Key=path_info['key'])
def md5s_to_path_infos(self, md5s):
return [{'scheme': self.scheme,
'bucket': self.bucket,
'key': posixpath.join(self.prefix, md5[0:2], md5[2:])} for md5 in md5s]
'key': posixpath.join(self.prefix,
md5[0:2], md5[2:])} for md5 in md5s]
def exists(self, path_infos):
ret = []
session = boto3.session.Session()
s3 = session.client('s3')
def _all_keys(self):
s3 = self.s3
keys = []
kwargs = {'Bucket': self.bucket,
@@ -141,6 +224,17 @@ class RemoteS3(RemoteBase):
kwargs['ContinuationToken'] = token
return keys
def exists(self, path_infos):
# NOTE: We mostly use exists() method when filtering a bulk of cache
# files to decide if we need to download/upload them and in s3
# list_objects_v2() is much-much faster than trying to check keys
# one-by-one.
ret = []
keys = self._all_keys()
for path_info in path_infos:
exists = False
if path_info['key'] in keys:
@@ -149,83 +243,90 @@ class RemoteS3(RemoteBase):
return ret
def upload(self, paths, path_infos, names=None):
assert isinstance(paths, list)
assert isinstance(path_infos, list)
assert len(paths) == len(path_infos)
if not names:
names = len(paths) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(paths)
def upload(self, from_infos, to_infos, names=None):
names = self._verify_path_args(to_infos, from_infos, names)
session = boto3.session.Session()
s3 = session.client('s3')
s3 = self.s3
for path, path_info, name in zip(paths, path_infos, names):
if path_info['scheme'] != 's3':
for from_info, to_info, name in zip(from_infos, to_infos, names):
if to_info['scheme'] != 's3':
raise NotImplementedError
Logger.debug("Uploading '{}' to '{}/{}'".format(path,
path_info['bucket'],
path_info['key']))
if from_info['scheme'] != 'local':
raise NotImplementedError
Logger.debug("Uploading '{}' to '{}/{}'".format(from_info['path'],
to_info['bucket'],
to_info['key']))
if not name:
name = os.path.basename(path)
name = os.path.basename(from_info['path'])
total = os.path.getsize(path)
total = os.path.getsize(from_info['path'])
cb = Callback(name, total)
try:
s3.upload_file(path, path_info['bucket'], path_info['key'], Callback=cb)
s3.upload_file(from_info['path'],
to_info['bucket'],
to_info['key'],
Callback=cb)
except Exception as exc:
Logger.error("Failed to upload '{}'".format(path), exc)
msg = "Failed to upload '{}'".format(from_info['path'])
Logger.error(msg, exc)
continue
progress.finish_target(name)
def download(self, path_infos, fnames, no_progress_bar=False, names=None):
assert isinstance(fnames, list)
assert isinstance(path_infos, list)
assert len(fnames) == len(path_infos)
if not names:
names = len(fnames) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(fnames)
def download(self,
from_infos,
to_infos,
no_progress_bar=False,
names=None):
names = self._verify_path_args(from_infos, to_infos, names)
session = boto3.session.Session()
s3 = session.client('s3')
s3 = self.s3
for fname, path_info, name in zip(fnames, path_infos, names):
if path_info['scheme'] != 's3':
for to_info, from_info, name in zip(to_infos, from_infos, names):
if from_info['scheme'] != 's3':
raise NotImplementedError
Logger.debug("Downloading '{}/{}' to '{}'".format(path_info['bucket'],
path_info['key'],
fname))
if to_info['scheme'] == 's3':
self._copy(from_info, to_info, s3=s3)
continue
tmp_file = self.tmp_file(fname)
if to_info['scheme'] != 'local':
raise NotImplementedError
msg = "Downloading '{}/{}' to '{}'".format(from_info['bucket'],
from_info['key'],
to_info['path'])
Logger.debug(msg)
tmp_file = self.tmp_file(to_info['path'])
if not name:
name = os.path.basename(fname)
name = os.path.basename(to_info['path'])
if no_progress_bar:
cb = None
else:
total = s3.head_object(Bucket=path_info['bucket'],
Key=path_info['key'])['ContentLength']
total = s3.head_object(Bucket=from_info['bucket'],
Key=from_info['key'])['ContentLength']
cb = Callback(name, total)
self._makedirs(fname)
self._makedirs(to_info['path'])
try:
s3.download_file(path_info['bucket'], path_info['key'], tmp_file, Callback=cb)
s3.download_file(from_info['bucket'],
from_info['key'],
tmp_file,
Callback=cb)
except Exception as exc:
Logger.error("Failed to download '{}/{}'".format(path_info['bucket'],
path_info['key']), exc)
msg = "Failed to download '{}/{}'".format(from_info['bucket'],
from_info['key'])
Logger.error(msg, exc)
return
os.rename(tmp_file, fname)
os.rename(tmp_file, to_info['path'])
if not no_progress_bar:
progress.finish_target(name)
@@ -235,16 +336,22 @@ class RemoteS3(RemoteBase):
return posixpath.dirname(relpath) + posixpath.basename(relpath)
def _all(self):
objects = self.s3.Bucket(self.bucket).objects.filter(Prefix=self.prefix)
return [self._path_to_etag(obj.key) for obj in objects]
keys = self._all_keys()
return [self._path_to_etag(key) for key in keys]
def gc(self, checksum_infos):
used_etags = [info[self.PARAM_ETAG] for info in checksum_infos]
def gc(self, cinfos):
used_etags = [info[self.PARAM_ETAG] for info in cinfos['s3']]
used_etags += [info[RemoteLOCAL.PARAM_MD5] for info in cinfos['local']]
removed = False
for etag in self._all():
if etag in used_etags:
continue
path_info = {'scheme': 's3',
'key': posixpath.join(self.prefix, etag[0:2], etag[2:]),
'key': posixpath.join(self.prefix,
etag[0:2], etag[2:]),
'bucket': self.bucket}
self.remove(path_info)
removed = True
return removed
+229 -75
View File
@@ -1,7 +1,5 @@
import os
import errno
import getpass
import filecmp
import posixpath
try:
@@ -10,9 +8,9 @@ except ImportError:
paramiko = None
from dvc.logger import Logger
from dvc.utils import copyfile, file_md5
from dvc.progress import progress
from dvc.remote.base import RemoteBase
from dvc.remote.local import RemoteLOCAL
from dvc.config import Config
from dvc.exceptions import DvcException
@@ -42,43 +40,72 @@ def create_cb(name):
class RemoteSSH(RemoteBase):
scheme = 'ssh'
#NOTE: temporarily only absolute paths are allowed
# NOTE: temporarily only absolute paths are allowed
REGEX = r'^ssh://((?P<user>.*)@)?(?P<host>[^/]*):(?P<path>/.*)$'
REQUIRES = {'paramiko': paramiko}
JOBS = 4
PARAM_MD5 = 'md5'
DEFAULT_PORT = 22
TIMEOUT = 1800
def __init__(self, project, config):
self.project = project
self.url = config.get(Config.SECTION_REMOTE_URL, '/')
self.host = self.group('host')
self.user = self.group('user')
if not self.user:
self.user = config.get(Config.SECTION_REMOTE_USER, getpass.getuser())
self.user = config.get(Config.SECTION_REMOTE_USER,
getpass.getuser())
self.prefix = self.group('path')
self.port = config.get(Config.SECTION_REMOTE_PORT, self.DEFAULT_PORT)
self.keyfile = config.get(Config.SECTION_REMOTE_KEY_FILE, None)
self.timeout = config.get(Config.SECTION_REMOTE_TIMEOUT, self.TIMEOUT)
self.password = config.get(Config.SECTION_REMOTE_PASSWORD, None)
self.ask_password = config.get(Config.SECTION_REMOTE_ASK_PASSWORD,
False)
def md5s_to_path_infos(self, md5s):
return [{'scheme': 'ssh',
'host': self.host,
'user': self.user,
'path': posixpath.join(self.prefix, md5[0:2], md5[2:])} for md5 in md5s]
'port': self.port,
'path': posixpath.join(self.prefix,
md5[0:2], md5[2:])} for md5 in md5s]
def ssh(self, host=None, user=None):
Logger.debug("Establishing ssh connection with '{}' as user '{}'".format(host, user))
def ssh(self, host=None, user=None, port=None):
msg = "Establishing ssh connection with '{}' " \
"through port '{}' as user '{}'"
Logger.debug(msg.format(host, port, user))
ssh = paramiko.SSHClient()
ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, username=user)
if self.ask_password and self.password is None:
msg = "host '{}' port '{}' user '{}'".format(host, port, user)
self.password = self.project.prompt.prompt_password(msg)
ssh.connect(host,
username=user,
port=port,
key_filename=self.keyfile,
timeout=self.timeout,
password=self.password)
return ssh
def exists(self, path_infos):
ret = []
ssh = self.ssh(host=self.host, user=self.user)
stdout = self._exec(ssh, 'find {} -type f -follow -print'.format(self.prefix))
ssh = self.ssh(host=self.host,
user=self.user,
port=self.port)
cmd = 'find {} -type f -follow -print'.format(self.prefix)
stdout = self._exec(ssh, cmd)
plist = stdout.split()
ssh.close()
@@ -92,15 +119,55 @@ class RemoteSSH(RemoteBase):
def _exec(self, ssh, cmd):
stdin, stdout, stderr = ssh.exec_command(cmd)
channel = stdout.channel
stdin.close()
channel.shutdown_write()
stdout_chunks = []
stderr_chunks = []
while (not channel.closed
or channel.recv_ready()
or channel.recv_stderr_ready()):
import select
got_chunk = False
readq, _, _ = select.select([stdout.channel], [], [], self.timeout)
for c in readq:
if c.recv_ready():
stdout_chunks.append(stdout.channel.recv(len(c.in_buffer)))
got_chunk = True
if c.recv_stderr_ready():
stderr_len = len(c.in_stderr_buffer)
s = stderr.channel.recv_stderr(stderr_len)
stderr_chunks.append(s)
got_chunk = True
if not got_chunk \
and stdout.channel.exit_status_ready() \
and not stderr.channel.recv_stderr_ready() \
and not stdout.channel.recv_ready():
stdout.channel.shutdown_read()
stdout.channel.close()
break
stdout.close()
stderr.close()
if stdout.channel.recv_exit_status() != 0:
DvcException('SSH command \'{}\' failed: {}'.format(cmd, stderr.read()))
return stdout.read().decode('utf-8')
err = b''.join(stderr_chunks).decode('utf-8')
msg = 'SSH command \'{}\' failed: {}'.format(cmd, err)
raise DvcException(msg)
return b''.join(stdout_chunks).decode('utf-8')
def md5(self, path_info):
if path_info['scheme'] != 'ssh':
raise NotImplementedError
ssh = self.ssh(host=path_info['host'], user=path_info['user'])
ssh = self.ssh(host=path_info['host'],
user=path_info['user'],
port=path_info['port'])
# Use different md5 commands depending on os
stdout = self._exec(ssh, 'uname').strip()
@@ -111,7 +178,8 @@ class RemoteSSH(RemoteBase):
md5cmd = 'md5sum'
index = 0
else:
raise DvcException('\'{}\' is not supported as a remote'.format(stdout))
msg = '\'{}\' is not supported as a remote'.format(stdout)
raise DvcException(msg)
stdout = self._exec(ssh, '{} {}'.format(md5cmd, path_info['path']))
md5 = stdout.split()[index]
@@ -121,20 +189,23 @@ class RemoteSSH(RemoteBase):
return md5
def cp(self, from_info, to_info):
def cp(self, from_info, to_info, ssh=None):
if from_info['scheme'] != 'ssh' or to_info['scheme'] != 'ssh':
raise NotImplementedError
assert from_info['host'] == to_info['host']
assert from_info['user'] == to_info['user']
ssh = self.ssh(host=from_info['host'], user=from_info['user'])
s = ssh if ssh else self.ssh(host=from_info['host'],
user=from_info['user'],
port=from_info['port'])
dname = posixpath.dirname(to_info['path'])
self._exec(ssh, 'mkdir -p {}'.format(dname))
self._exec(ssh, 'cp {} {}'.format(from_info['path'], to_info['path']))
self._exec(s, 'mkdir -p {}'.format(dname))
self._exec(s, 'cp {} {}'.format(from_info['path'], to_info['path']))
ssh.close()
if not ssh:
s.close()
def save_info(self, path_info):
if path_info['scheme'] != 'ssh':
@@ -142,6 +213,43 @@ class RemoteSSH(RemoteBase):
return {self.PARAM_MD5: self.md5(path_info)}
@staticmethod
def to_string(path_info):
return "{}://{}@{}:{}".format(path_info['scheme'],
path_info['user'],
path_info['host'],
path_info['path'])
def changed_cache(self, md5):
cache = {}
cache['scheme'] = 'ssh'
cache['host'] = self.host
cache['port'] = self.port
cache['user'] = self.user
cache['path'] = posixpath.join(self.prefix, md5[0:2], md5[2:])
if {self.PARAM_MD5: md5} != self.save_info(cache):
if self.exists([cache])[0]:
msg = 'Corrupted cache file {}'
Logger.warn(msg.format(self.to_string(cache)))
self.remove(cache)
return True
return False
def changed(self, path_info, checksum_info):
if not self.exists([path_info])[0]:
return True
md5 = checksum_info.get(self.PARAM_MD5, None)
if md5 is None:
return True
if self.changed_code(md5):
return True
return checksum_info != self.save_info(path_info)
def save(self, path_info):
if path_info['scheme'] != 'ssh':
raise NotImplementedError
@@ -162,6 +270,25 @@ class RemoteSSH(RemoteBase):
if not md5:
return
if not self.changed(path_info, checksum_info):
msg = "Data '{}' didn't change."
Logger.info(msg.format(self.to_string(path_info)))
return
if self.changed_cache(md5):
msg = "Cache '{}' not found. File '{}' won't be created."
Logger.warn(msg.format(md5, self.to_string(path_info)))
return
if self.exists([path_info])[0]:
msg = "Data '{}' exists. Removing before checkout."
Logger.warn(msg.format(self.to_string(path_info)))
self.remove(path_info)
return
msg = "Checking out '{}' with cache '{}'."
Logger.info(msg.format(self.to_string(path_info), md5))
src = path_info.copy()
src['path'] = posixpath.join(self.prefix, md5[0:2], md5[2:])
@@ -175,80 +302,97 @@ class RemoteSSH(RemoteBase):
path_info['host'],
path_info['path']))
ssh = self.ssh(host=path_info['host'], user=path_info['user'])
ssh = self.ssh(host=path_info['host'],
user=path_info['user'],
port=path_info['port'])
ssh.open_sftp().remove(path_info['path'])
ssh.close()
def download(self, path_infos, paths, no_progress_bar=False, names=None):
assert isinstance(paths, list)
assert isinstance(path_infos, list)
assert len(paths) == len(path_infos)
if not names:
names = len(paths) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(paths)
def download(self,
from_infos,
to_infos,
no_progress_bar=False,
names=None):
names = self._verify_path_args(from_infos, to_infos, names)
ssh = self.ssh(host=path_infos[0]['host'], user=path_infos[0]['user'])
ssh = self.ssh(host=from_infos[0]['host'],
user=from_infos[0]['user'],
port=from_infos[0]['port'])
for path, path_info, name in zip(paths, path_infos, names):
if path_info['scheme'] != 'ssh':
for to_info, from_info, name in zip(to_infos, from_infos, names):
if from_info['scheme'] != 'ssh':
raise NotImplementedError
Logger.debug("Downloading '{}/{}' to '{}'".format(path_info['host'],
path_info['path'],
path))
if not name:
name = os.path.basename(path)
self._makedirs(path)
tmp_file = self.tmp_file(path)
try:
ssh.open_sftp().get(path_info['path'], tmp_file, callback=create_cb(name))
except Exception as exc:
Logger.error("Failed to download '{}/{}' to '{}'".format(path_info['host'],
path_info['path'],
path), exc)
if to_info['scheme'] == 'ssh':
assert from_info['host'] == to_info['host']
assert from_info['port'] == to_info['port']
assert from_info['user'] == to_info['user']
self.cp(from_info, to_info, ssh=ssh)
continue
os.rename(tmp_file, path)
if to_info['scheme'] != 'local':
raise NotImplementedError
msg = "Downloading '{}/{}' to '{}'".format(from_info['host'],
from_info['path'],
to_info['path'])
Logger.debug(msg)
if not name:
name = os.path.basename(to_info['path'])
self._makedirs(to_info['path'])
tmp_file = self.tmp_file(to_info['path'])
try:
ssh.open_sftp().get(from_info['path'],
tmp_file,
callback=create_cb(name))
except Exception as exc:
msg = "Failed to download '{}/{}' to '{}'"
Logger.error(msg.format(from_info['host'],
from_info['path'],
to_info['path']), exc)
continue
os.rename(tmp_file, to_info['path'])
progress.finish_target(name)
ssh.close()
def upload(self, paths, path_infos, names=None):
assert isinstance(paths, list)
assert isinstance(path_infos, list)
assert len(paths) == len(path_infos)
if not names:
names = len(paths) * [None]
else:
assert isinstance(names, list)
assert len(names) == len(paths)
def upload(self, from_infos, to_infos, names=None):
names = self._verify_path_args(to_infos, from_infos, names)
ssh = self.ssh(host=path_infos[0]['host'], user=path_infos[0]['user'])
ssh = self.ssh(host=to_infos[0]['host'],
user=to_infos[0]['user'],
port=to_infos[0]['port'])
sftp = ssh.open_sftp()
for path, path_info, name in zip(paths, path_infos, names):
if path_info['scheme'] != 'ssh':
for from_info, to_info, name in zip(from_infos, to_infos, names):
if to_info['scheme'] != 'ssh':
raise NotImplementedError
Logger.debug("Uploading '{}' to '{}/{}'".format(path,
path_info['host'],
path_info['path']))
if from_info['scheme'] != 'local':
raise NotImplementedError
Logger.debug("Uploading '{}' to '{}/{}'".format(from_info['path'],
to_info['host'],
to_info['path']))
if not name:
name = os.path.basename(path)
name = os.path.basename(from_info['path'])
dname = posixpath.dirname(path_info['path'])
dname = posixpath.dirname(to_info['path'])
self._exec(ssh, 'mkdir -p {}'.format(dname))
try:
sftp.put(path, path_info['path'], callback=create_cb(name))
sftp.put(from_info['path'],
to_info['path'],
callback=create_cb(name))
except Exception as exc:
Logger.error("Failed to upload '{}' to '{}/{}'".format(path,
path_info['host'],
path_info['path']), exc)
msg = "Failed to upload '{}' to '{}/{}'"
Logger.error(msg.format(from_info['path'],
to_info['host'],
to_info['path'], exc))
continue
progress.finish_target(name)
@@ -261,21 +405,31 @@ class RemoteSSH(RemoteBase):
return posixpath.dirname(relpath) + posixpath.basename(relpath)
def _all_md5s(self):
ssh = self.ssh(host=self.host, user=self.user)
stdout = self._exec(ssh, 'find {} -type f -follow -print'.format(self.prefix))
ssh = self.ssh(host=self.host,
user=self.user,
port=self.port)
cmd = 'find {} -type f -follow -print'.format(self.prefix)
stdout = self._exec(ssh, cmd)
flist = stdout.split()
ssh.close()
return [self._path_to_md5(path) for path in flist]
def gc(self, checksum_infos):
used_md5s = [info[self.PARAM_MD5] for info in checksum_infos]
def gc(self, cinfos):
used = [info[self.PARAM_MD5] for info in cinfos['ssh']]
used += [info[RemoteLOCAL.PARAM_MD5] for info in cinfos['local']]
removed = False
for md5 in self._all_md5s():
if md5 in used_md5s:
if md5 in used:
continue
path_info = {'scheme': 'ssh',
'user': self.user,
'host': self.host,
'path': posixpath.join(self.prefix, md5[0:2], md5[2:])}
'port': self.port,
'path': posixpath.join(self.prefix,
md5[0:2], md5[2:])}
self.remove(path_info)
removed = True
return removed
+92 -27
View File
@@ -2,6 +2,7 @@ import os
from dvc.exceptions import DvcException
from dvc.logger import Logger
from dvc.utils import fix_env
class SCMError(DvcException):
@@ -13,13 +14,18 @@ class FileNotInRepoError(DvcException):
class Base(object):
def __init__(self, root_dir=os.curdir):
self.root_dir = os.path.abspath(os.path.realpath(root_dir))
def __init__(self, root_dir=os.curdir, project=None):
self.project = project
self.root_dir = root_dir
@staticmethod
def is_repo(root_dir):
return True
@staticmethod
def is_submodule(root_dir):
return True
def ignore(self, path):
pass
@@ -44,17 +50,34 @@ class Base(object):
def branch(self, branch):
pass
def brancher(self, branches=None, all_branches=False):
if not branches and not all_branches:
def brancher(self,
branches=None,
all_branches=False,
tags=None,
all_tags=False):
if not branches and not all_branches \
and not tags and not all_tags:
yield ''
return
saved = self.active_branch()
if not branches:
branches = self.list_branches() if all_branches else [saved]
for branch in branches:
self.checkout(branch)
yield branch
revs = []
if branches is not None:
revs.extend(branches)
elif all_branches:
revs.extend(self.list_branches())
elif tags is not None:
revs.extend(tags)
elif all_tags:
revs.extend(self.list_tags())
else:
revs.extend([saved])
for rev in revs:
self.checkout(rev)
yield rev
self.checkout(saved)
def untracked_files(self):
@@ -69,6 +92,9 @@ class Base(object):
def list_branches(self):
pass
def list_tags(self):
pass
def install(self):
pass
@@ -77,28 +103,46 @@ class Git(Base):
GITIGNORE = '.gitignore'
GIT_DIR = '.git'
def __init__(self, root_dir=os.curdir):
super(Git, self).__init__(root_dir)
def __init__(self, root_dir=os.curdir, project=None):
super(Git, self).__init__(root_dir, project=project)
import git
from git.exc import InvalidGitRepositoryError
try:
self.repo = git.Repo(root_dir)
except InvalidGitRepositoryError:
raise SCMError('{} is not a git repository'.format(root_dir))
msg = '{} is not a git repository'
raise SCMError(msg.format(root_dir))
# NOTE: fixing LD_LIBRARY_PATH for binary built by PyInstaller.
# http://pyinstaller.readthedocs.io/en/stable/runtime-information.html
env = fix_env(None)
lp = env.get('LD_LIBRARY_PATH', None)
self.repo.git.update_environment(LD_LIBRARY_PATH=lp)
@staticmethod
def is_repo(root_dir):
git_dir = os.path.join(root_dir, Git.GIT_DIR)
return os.path.isdir(git_dir)
return os.path.isdir(Git._get_git_dir(root_dir))
@staticmethod
def is_submodule(root_dir):
return os.path.isfile(Git._get_git_dir(root_dir))
@staticmethod
def _get_git_dir(root_dir):
return os.path.join(root_dir, Git.GIT_DIR)
@property
def dir(self):
return self.repo.git_dir
def ignore_file(self):
return self.GITIGNORE
def _get_gitignore(self, path):
assert os.path.isabs(path)
entry = os.path.basename(path)
gitignore = os.path.join(os.path.abspath(os.path.dirname(path)),
self.GITIGNORE)
gitignore = os.path.abspath(os.path.realpath(gitignore))
gitignore = os.path.join(os.path.dirname(path), self.GITIGNORE)
if not gitignore.startswith(self.root_dir):
raise FileNotInRepoError(path)
@@ -111,15 +155,24 @@ class Git(Base):
ignore_list = []
if os.path.exists(gitignore):
ignore_list = open(gitignore, 'r').readlines()
filtered = list(filter(lambda x: x.strip() == entry.strip(), ignore_list))
filtered = list(filter(lambda x: x.strip() == entry.strip(),
ignore_list))
if len(filtered) != 0:
return
msg = "Adding '{}' to '{}'.".format(os.path.relpath(path),
os.path.relpath(gitignore))
Logger.info(msg)
content = entry
if len(ignore_list) > 0:
content = '\n' + content
open(gitignore, 'a').write(content)
with open(gitignore, 'a') as fd:
fd.write(content)
if self.project is not None:
self.project._files_to_git_add.append(os.path.relpath(gitignore))
def ignore_remove(self, path):
entry, gitignore = self._get_gitignore(path)
@@ -135,14 +188,19 @@ class Git(Base):
with open(gitignore, 'w') as fd:
fd.writelines(filtered)
if self.project is not None:
self.project._files_to_git_add.append(os.path.relpath(gitignore))
def add(self, paths):
# NOTE: GitPython is not currently able to handle index version >= 3.
# See https://github.com/iterative/dvc/issues/610 for more details.
try:
self.repo.index.add(paths)
except AssertionError as exc:
msg = 'Failed to add \'{}\' to git. You can add those files manually using \'git add\'. '
msg += 'See \'https://github.com/iterative/dvc/issues/610\' for more details.'
msg = 'Failed to add \'{}\' to git. You can add those files '
msg += 'manually using \'git add\'. '
msg += 'See \'https://github.com/iterative/dvc/issues/610\' '
msg += 'for more details.'
Logger.error(msg.format(str(paths)), exc)
def commit(self, msg):
@@ -170,17 +228,24 @@ class Git(Base):
def list_branches(self):
return [h.name for h in self.repo.heads]
def list_tags(self):
return [t.name for t in self.repo.tags]
def install(self):
hook = os.path.join(self.root_dir, self.GIT_DIR, 'hooks', 'post-checkout')
hook = os.path.join(self.root_dir,
self.GIT_DIR,
'hooks',
'post-checkout')
if os.path.isfile(hook):
raise SCMError('Git hook \'{}\' already exists.'.format(os.path.relpath(hook)))
msg = 'Git hook \'{}\' already exists.'
raise SCMError(msg.format(os.path.relpath(hook)))
with open(hook, 'w+') as fd:
fd.write('#!/bin/sh\nexec dvc checkout\n')
os.chmod(hook, 0o777)
def SCM(root_dir=os.curdir, no_scm=False):
if Git.is_repo(root_dir):
return Git(root_dir)
def SCM(root_dir, no_scm=False, project=None):
if Git.is_repo(root_dir) or Git.is_submodule(root_dir):
return Git(root_dir, project=project)
return Base(root_dir)
return Base(root_dir, project=project)
-17
View File
@@ -1,17 +0,0 @@
import signal
import threading
from dvc.logger import Logger
class SignalHandler(object):
def __enter__(self):
if isinstance(threading.current_thread(), threading._MainThread):
self.old_handler = signal.signal(signal.SIGINT, self.handler)
def handler(self, sig, frame):
Logger.debug('Ignoring SIGINT during critical parts of code...')
def __exit__(self, type, value, traceback):
if isinstance(threading.current_thread(), threading._MainThread):
signal.signal(signal.SIGINT, self.old_handler)
+233 -70
View File
@@ -1,14 +1,14 @@
import os
import yaml
import itertools
import posixpath
import subprocess
import schema
from schema import Schema, SchemaError, Optional, Or, And
import dvc.dependency as dependency
import dvc.output as output
from dvc.exceptions import DvcException
from dvc.logger import Logger
from dvc.utils import dict_md5
from dvc.utils import dict_md5, fix_env
class StageCmdFailedError(DvcException):
@@ -18,8 +18,39 @@ class StageCmdFailedError(DvcException):
class StageFileFormatError(DvcException):
def __init__(self):
super(StageFileFormatError, self).__init__('Stage file format error')
def __init__(self, fname, e):
msg = "Stage file '{}' format error: {}".format(fname, str(e))
super(StageFileFormatError, self).__init__(msg)
class StageFileDoesNotExistError(DvcException):
def __init__(self, fname):
msg = "'{}' does not exist.".format(fname)
super(StageFileDoesNotExistError, self).__init__(msg)
class StageFileIsNotDvcFileError(DvcException):
def __init__(self, fname):
msg = "'{}' is not a dvc file".format(fname)
super(StageFileIsNotDvcFileError, self).__init__(msg)
class StageFileBadNameError(DvcException):
def __init__(self, msg):
super(StageFileBadNameError, self).__init__(msg)
class MissingDep(DvcException):
def __init__(self, deps):
assert len(deps) > 0
if len(deps) > 1:
dep = 'dependencies'
else:
dep = 'dependency'
msg = u'missing {}: {}'.format(dep, ', '.join(map(str, deps)))
super(MissingDep, self).__init__(msg)
class MissingDataSource(DvcException):
@@ -45,14 +76,22 @@ class Stage(object):
PARAM_LOCKED = 'locked'
SCHEMA = {
schema.Optional(PARAM_MD5): schema.Or(str, None),
schema.Optional(PARAM_CMD): schema.Or(str, None),
schema.Optional(PARAM_DEPS): schema.Or(schema.And(list, schema.Schema([dependency.SCHEMA])), None),
schema.Optional(PARAM_OUTS): schema.Or(schema.And(list, schema.Schema([output.SCHEMA])), None),
schema.Optional(PARAM_LOCKED): bool,
Optional(PARAM_MD5): Or(str, None),
Optional(PARAM_CMD): Or(str, None),
Optional(PARAM_DEPS): Or(And(list, Schema([dependency.SCHEMA])), None),
Optional(PARAM_OUTS): Or(And(list, Schema([output.SCHEMA])), None),
Optional(PARAM_LOCKED): bool,
}
def __init__(self, project, path=None, cmd=None, cwd=None, deps=[], outs=[], md5=None, locked=False):
def __init__(self,
project,
path=None,
cmd=None,
cwd=os.curdir,
deps=[],
outs=[],
md5=None,
locked=False):
self.project = project
self.path = path
self.cmd = cmd
@@ -70,28 +109,31 @@ class Stage(object):
def is_data_source(self):
return self.cmd is None
@staticmethod
def is_stage_filename(path):
if not path.endswith(Stage.STAGE_FILE_SUFFIX) \
and os.path.basename(path) != Stage.STAGE_FILE:
return False
return True
@staticmethod
def is_stage_file(path):
if not os.path.isfile(path):
return False
if not path.endswith(Stage.STAGE_FILE_SUFFIX) and os.path.basename(path) != Stage.STAGE_FILE:
return False
return True
return Stage.is_stage_filename(path)
def changed_md5(self):
md5 = self.dumpd().get(self.PARAM_MD5, None)
# backward compatibility
if self.md5 == None:
if self.md5 is None:
return False
if self.md5 and md5 and self.md5 == md5:
return False
msg = "Dvc file '{}' md5 changed(expected '{}', actual '{}')"
self.project.logger.debug(msg.format(self.relpath, self.md5, md5))
return True
@property
@@ -102,31 +144,53 @@ class Stage(object):
def is_import(self):
return not self.cmd and \
len(self.deps) == 1 and \
len(self.outs) == 1 and \
self.outs[0].path_info['scheme'] == 'local'
len(self.outs) == 1
def changed(self):
def changed(self, print_info=False):
ret = False
if print_info:
log = self.project.logger.info
else:
log = self.project.logger.debug
if self.is_callback:
msg = "Dvc file '{}' is a 'callback' stage (has a command and " \
"no dependencies) and thus always considered as changed."
self.project.logger.warn(msg.format(self.relpath))
ret = True
if self.locked:
entries = self.outs
else:
entries = itertools.chain(self.outs, self.deps)
for entry in entries:
if entry.changed():
if not self.locked:
for dep in self.deps:
if not dep.changed():
continue
if print_info:
msg = "Dependency '{}' of '{}' changed."
log(msg.format(dep, self.relpath))
ret = True
for out in self.outs:
if not out.changed():
continue
if print_info:
msg = "Output '{}' of '{}' changed."
log(msg.format(out, self.relpath))
ret = True
if self.changed_md5():
if print_info:
msg = "Dvc file '{}' changed."
log(msg.format(self.relpath))
ret = True
if ret:
self.project.logger.debug(u'Dvc file \'{}\' changed'.format(self.relpath))
msg = "Stage '{}' changed.".format(self.relpath)
color = 'yellow'
else:
self.project.logger.debug(u'Dvc file \'{}\' didn\'t change'.format(self.relpath))
msg = "Stage '{}' didn't change.".format(self.relpath)
color = 'green'
log(Logger.colorize(msg, color))
return ret
@@ -138,34 +202,39 @@ class Stage(object):
self.remove_outs(ignore_remove=True)
os.unlink(self.path)
def reproduce(self, force=False):
if not self.changed() and not force:
def reproduce(self, force=False, dry=False, interactive=False):
if not self.changed(print_info=True) and not force:
return None
if (self.cmd or self.is_import) and not self.locked:
if (self.cmd or self.is_import) and not self.locked and not dry:
# Removing outputs only if we actually have command to reproduce
self.remove_outs(ignore_remove=False)
msg = "Going to reproduce '{}'. Are you sure you want to continue?"
msg = msg.format(self.relpath)
if interactive \
and not self.project.prompt.prompt(msg):
raise DvcException('Reproduction aborted by the user')
self.project.logger.info(u'Reproducing \'{}\''.format(self.relpath))
self.run()
self.run(dry=dry)
self.project.logger.debug(u'\'{}\' was reproduced'.format(self.relpath))
msg = u'\'{}\' was reproduced'.format(self.relpath)
self.project.logger.debug(msg)
return self
@staticmethod
def validate(d):
def validate(d, fname=None):
try:
schema.Schema(Stage.SCHEMA).validate(d)
except schema.SchemaError as exc:
Logger.debug(str(exc))
raise StageFileFormatError()
Schema(Stage.SCHEMA).validate(d)
except SchemaError as exc:
raise StageFileFormatError(fname, exc)
@staticmethod
def loadd(project, d, path):
Stage.validate(d)
Stage.validate(d, fname=os.path.relpath(path))
path = os.path.abspath(path)
cwd = os.path.dirname(path)
cmd = d.get(Stage.PARAM_CMD, None)
@@ -184,6 +253,28 @@ class Stage(object):
return stage
@classmethod
def _stage_fname_cwd(cls, fname, cwd, outs, add):
if fname and cwd:
return (fname, cwd)
if not outs:
return (cls.STAGE_FILE, cwd if cwd else os.getcwd())
out = outs[0]
if out.path_info['scheme'] == 'local':
path = os.path
else:
path = posixpath
if not fname:
fname = path.basename(out.path) + cls.STAGE_FILE_SUFFIX
if not cwd or (add and out.is_local):
cwd = path.dirname(out.path)
return (fname, cwd)
@staticmethod
def loads(project=None,
cmd=None,
@@ -193,25 +284,69 @@ class Stage(object):
metrics_no_cache=[],
fname=None,
cwd=os.curdir,
locked=False):
cwd = os.path.abspath(cwd)
path = os.path.join(cwd, fname)
locked=False,
add=False,
overwrite=True):
stage = Stage(project=project,
path=path,
cmd=cmd,
cwd=cwd,
cmd=cmd,
locked=locked)
stage.outs = output.loads_from(stage, outs, use_cache=True)
stage.outs += output.loads_from(stage, outs_no_cache, use_cache=False)
stage.outs += output.loads_from(stage, metrics_no_cache, use_cache=False, metric=True)
stage.outs += output.loads_from(stage, metrics_no_cache,
use_cache=False, metric=True)
stage.deps = dependency.loads_from(stage, deps)
if fname is not None and os.path.basename(fname) != fname:
msg = "Stage file name '{}' should not contain subdirectories. " \
"Use '-c|--cwd' to change location of the stage file."
raise StageFileBadNameError(msg.format(fname))
fname, cwd = Stage._stage_fname_cwd(fname, cwd, stage.outs, add=add)
cwd = os.path.abspath(cwd)
path = os.path.join(cwd, fname)
if os.path.exists(path):
relpath = os.path.relpath(path)
msg = "'{}' already exists. " \
"Do you wish to run the command and overwrite it?"
if not overwrite \
and not project.prompt.prompt(msg.format(relpath), False):
raise DvcException("'{}' already exists".format(relpath))
stage.cwd = cwd
stage.path = path
return stage
@staticmethod
def _check_dvc_filename(fname):
if not Stage.is_stage_filename(fname):
msg = "Bad stage filename '{}'. Stage files should be named " \
"'Dvcfile' or have a '.dvc' suffix(e.g. '{}.dvc')."
raise StageFileBadNameError(msg.format(os.path.relpath(fname),
os.path.basename(fname)))
@staticmethod
def _check_dvc_file(fname):
sname = fname + Stage.STAGE_FILE_SUFFIX
if Stage.is_stage_file(sname):
Logger.info("Do you mean '{}'?".format(sname))
@staticmethod
def load(project, fname):
if not os.path.exists(fname):
Stage._check_dvc_file(fname)
raise StageFileDoesNotExistError(fname)
Stage._check_dvc_filename(fname)
if not Stage.is_stage_file(fname):
Stage._check_dvc_file(fname)
raise StageFileIsNotDvcFileError(fname)
with open(fname, 'r') as fd:
return Stage.loadd(project, yaml.safe_load(fd), fname)
@@ -220,7 +355,7 @@ class Stage(object):
outs = [x.dumpd() for x in self.outs]
ret = {}
if self.cmd != None:
if self.cmd is not None:
ret[Stage.PARAM_CMD] = self.cmd
if len(deps):
@@ -240,9 +375,16 @@ class Stage(object):
if not fname:
fname = self.path
self._check_dvc_filename(fname)
msg = "Saving information to '{}'.".format(os.path.relpath(fname))
Logger.info(msg)
with open(fname, 'w') as fd:
yaml.safe_dump(self.dumpd(), fd, default_flow_style=False)
self.project._files_to_git_add.append(os.path.relpath(fname))
def save(self):
for dep in self.deps:
dep.save()
@@ -250,36 +392,57 @@ class Stage(object):
for out in self.outs:
out.save()
def run(self):
def _check_missing_deps(self):
missing = []
for dep in self.deps:
if not dep.exists:
missing.append(dep)
if len(missing) > 0:
raise MissingDep(missing)
def run(self, dry=False):
if self.locked:
self.project.logger.info(u'Verifying outputs in locked stage \'{}\''.format(self.relpath))
self.check_missing_outputs()
msg = u'Verifying outputs in locked stage \'{}\''
self.project.logger.info(msg.format(self.relpath))
if not dry:
self.check_missing_outputs()
elif self.is_import:
msg = u'Importing \'{}\' -> \'{}\''
self.project.logger.info(msg.format(self.deps[0].path, self.outs[0].rel_path))
self.project.logger.info(msg.format(self.deps[0].path,
self.outs[0].path))
self.deps[0].download(self.outs[0].path)
if not dry:
self.deps[0].download(self.outs[0].path_info)
elif self.is_data_source:
self.project.logger.info(u'Verifying data sources in \'{}\''.format(self.relpath))
self.check_missing_outputs()
msg = u'Verifying data sources in \'{}\''.format(self.relpath)
self.project.logger.info(msg)
if not dry:
self.check_missing_outputs()
else:
self.project.logger.info(u'Running command:\n\t{}'.format(self.cmd))
msg = u'Running command:\n\t{}'.format(self.cmd)
self.project.logger.info(msg)
p = subprocess.Popen(self.cmd,
cwd=self.cwd,
shell=True,
env=os.environ,
executable=os.getenv('SHELL'))
p.communicate()
if p.returncode != 0:
raise StageCmdFailedError(self)
if not dry:
self._check_missing_deps()
p = subprocess.Popen(self.cmd,
cwd=self.cwd,
shell=True,
env=fix_env(os.environ),
executable=os.getenv('SHELL'))
p.communicate()
if p.returncode != 0:
raise StageCmdFailedError(self)
self.save()
if not dry:
self.save()
def check_missing_outputs(self):
missing_outs = [out.rel_path for out in self.outs if not os.path.exists(out.rel_path)]
if missing_outs:
raise MissingDataSource(missing_outs)
outs = [out for out in self.outs if not out.exists]
paths = [out.path if out.path_info['scheme'] != 'local' else
out.rel_path for out in outs]
if paths:
raise MissingDataSource(paths)
def checkout(self):
for out in self.outs:
+231 -167
View File
@@ -1,94 +1,76 @@
import os
import json
import time
import sqlite3
import nanotime
import threading
from dvc.lock import Lock
from dvc.config import Config
from dvc.system import System
from dvc.utils import file_md5, dict_md5, remove
from dvc.utils import file_md5, remove
from dvc.exceptions import DvcException
from dvc.signal_handler import SignalHandler
from dvc.logger import Logger
class StateEntry(object):
PARAM_MTIME = 'mtime'
PARAM_MD5 = 'md5'
def __init__(self, md5, mtime):
self.mtime = mtime
self.md5 = md5
def update(self, md5, mtime):
self.mtime = mtime
self.md5 = md5
@staticmethod
def loadd(d):
mtime = d[StateEntry.PARAM_MTIME]
md5 = d[StateEntry.PARAM_MD5]
return StateEntry(md5, mtime)
def dumpd(self):
return {
self.PARAM_MD5: self.md5,
self.PARAM_MTIME: self.mtime,
}
class StateDuplicateError(DvcException):
pass
class State(object):
VERSION = 1
STATE_FILE = 'state'
STATE_LOCK_FILE = 'state.lock'
STATE_TABLE = 'state'
STATE_TABLE_LAYOUT = "inode UNSIGNED INTEGER PRIMARY KEY, " \
"mtime TEXT NOT NULL, " \
"md5 TEXT NOT NULL, " \
"timestamp TEXT NOT NULL"
PARAM_RELPATH = 'relpath'
PARAM_MD5 = 'md5'
MD5_DIR_SUFFIX = '.dir'
STATE_INFO_TABLE = 'state_info'
STATE_INFO_TABLE_LAYOUT = 'count INTEGER'
STATE_INFO_ROW = 1
def __init__(self, project):
LINK_STATE_TABLE = 'link_state'
LINK_STATE_TABLE_LAYOUT = "path TEXT PRIMARY KEY, " \
"inode UNSIGNED INTEGER NOT NULL, " \
"mtime TEXT NOT NULL"
STATE_ROW_LIMIT = 10000000
STATE_ROW_CLEANUP_QUOTA = 50
def __init__(self, project, config):
self.project = project
self.dvc_dir = project.dvc_dir
self._lock = threading.Lock()
if self.dvc_dir:
self.state_file = os.path.join(self.dvc_dir, self.STATE_FILE)
self._lock_file = Lock(self.dvc_dir, self.STATE_LOCK_FILE)
else:
self.root_dir = project.root_dir
self.row_limit = 100
self.row_cleanup_quota = 50
c = config.get(Config.SECTION_STATE, {})
self.row_limit = c.get(Config.SECTION_STATE_ROW_LIMIT,
self.STATE_ROW_LIMIT)
self.row_cleanup_quota = c.get(Config.SECTION_STATE_ROW_CLEANUP_QUOTA,
self.STATE_ROW_CLEANUP_QUOTA)
if not self.dvc_dir:
self.state_file = None
self._lock_file = threading.Lock()
self._db = self.load()
return
self.state_file = os.path.join(self.dvc_dir, self.STATE_FILE)
self.db = None
self.c = None
self.inserts = 0
def __enter__(self):
self.load()
def __exit__(self, type, value, tb):
self.dump()
@staticmethod
def init(project):
return State(project)
def _collect_dir(self, dname):
dir_info = []
for root, dirs, files in os.walk(dname):
for fname in files:
path = os.path.join(root, fname)
relpath = os.path.relpath(path, dname)
# FIXME: we could've used `md5 = self.update(path, dump=False)` here,
# but it is around twice as slow(on ssd, don't know about hdd) for a
# directory with small files. What we could do here is introduce some
# kind of a limit for file size, after which we would actually register
# it in our state file.
md5 = file_md5(path)[0]
dir_info.append({self.PARAM_RELPATH: relpath, self.PARAM_MD5: md5})
md5 = dict_md5(dir_info) + self.MD5_DIR_SUFFIX
if self.project.cache.local.changed(md5):
self.project.cache.local.dump_dir_cache(md5, dir_info)
return (md5, dir_info)
def _collect(self, path):
if os.path.isdir(path):
return self._collect_dir(path)
return self.project.cache.local.collect_dir_cache(path)
else:
return (file_md5(path)[0], None)
@@ -100,136 +82,220 @@ class State(object):
return actual.split('.')[0] != md5.split('.')[0]
def load(self):
if not self.state_file or not os.path.isfile(self.state_file):
return {}
def _execute(self, cmd):
Logger.debug(cmd)
return self.c.execute(cmd)
with open(self.state_file, 'r') as fd:
def _load(self):
from dvc import VERSION
cmd = "PRAGMA user_version;"
self._execute(cmd)
ret = self.c.fetchall()
assert len(ret) == 1
assert len(ret[0]) == 1
assert isinstance(ret[0][0], int)
self.version = ret[0][0]
if self.version > self.VERSION:
msg = "You are using an old version '{}' of dvc that is using " \
"state file version '{}' which is not compatible with the " \
"state file version '{}' that is used in this projet. " \
"Please upgrade right now!"
raise DvcException(msg.format(VERSION,
self.VERSION,
self.version))
elif self.version < self.VERSION:
msg = "State file version '{}' is too old. " \
"Reformatting to the current version '{}'."
self.project.logger.warn(msg.format(self.version, self.VERSION))
cmd = "DROP TABLE IF EXISTS {};"
self._execute(cmd.format(self.STATE_TABLE))
self._execute(cmd.format(self.STATE_INFO_TABLE))
self._execute(cmd.format(self.LINK_STATE_TABLE))
# Check that the state file is indeed a database
cmd = "CREATE TABLE IF NOT EXISTS {} ({})"
self._execute(cmd.format(self.STATE_TABLE,
self.STATE_TABLE_LAYOUT))
self._execute(cmd.format(self.STATE_INFO_TABLE,
self.STATE_INFO_TABLE_LAYOUT))
self._execute(cmd.format(self.LINK_STATE_TABLE,
self.LINK_STATE_TABLE_LAYOUT))
cmd = "INSERT OR IGNORE INTO {} (count) SELECT 0 " \
"WHERE NOT EXISTS (SELECT * FROM {})"
self._execute(cmd.format(self.STATE_INFO_TABLE,
self.STATE_INFO_TABLE))
cmd = "PRAGMA user_version = {};"
self._execute(cmd.format(self.VERSION))
def load(self):
retries = 1
while True:
assert self.db is None
assert self.c is None
assert self.inserts == 0
self.db = sqlite3.connect(self.state_file)
self.c = self.db.cursor()
# Try loading once to check that the file is indeed a database
# and reformat it if it is not.
try:
return json.load(fd)
except ValueError as exc:
Logger.error('Failed to load \'{}\''.format(self.state_file), exc)
return {}
self._load()
return
except sqlite3.DatabaseError:
self.c.close()
self.db.close()
self.db = None
self.c = None
self.inserts = 0
if retries > 0:
os.unlink(self.state_file)
retries -= 1
else:
raise
def dump(self):
if not self.state_file:
return
assert self.db is not None
with SignalHandler():
with open(self.state_file, 'w+') as fd:
json.dump(self._db, fd)
cmd = "SELECT count from {} WHERE rowid={}"
self._execute(cmd.format(self.STATE_INFO_TABLE,
self.STATE_INFO_ROW))
ret = self.c.fetchall()
assert len(ret) == 1
assert len(ret[0]) == 1
count = ret[0][0] + self.inserts
if count > self.row_limit:
msg = "Cleaning up state. This might take a while."
self.project.logger.warn(msg)
delete = (count - self.row_limit)
delete += int(self.row_limit * (self.row_cleanup_quota/100.))
cmd = "DELETE FROM {} WHERE timestamp IN (" \
"SELECT timestamp FROM {} ORDER BY timestamp ASC LIMIT {});"
self._execute(cmd.format(self.STATE_TABLE,
self.STATE_TABLE,
delete))
self._execute("VACUUM")
cmd = "SELECT COUNT(*) FROM {}"
self._execute(cmd.format(self.STATE_TABLE))
ret = self.c.fetchall()
assert len(ret) == 1
assert len(ret[0]) == 1
count = ret[0][0]
cmd = "UPDATE {} SET count = {} WHERE rowid = {}"
self._execute(cmd.format(self.STATE_INFO_TABLE,
count,
self.STATE_INFO_ROW))
self.db.commit()
self.c.close()
self.db.close()
self.db = None
self.c = None
self.inserts = 0
@staticmethod
def mtime(path):
return str(int(nanotime.timestamp(os.path.getmtime(path))))
mtime = os.path.getmtime(path)
if os.path.isdir(path):
for root, dirs, files in os.walk(path):
for entry in list(dirs) + list(files):
p = os.path.join(root, entry)
m = os.path.getmtime(p)
if m > mtime:
mtime = m
return str(int(nanotime.timestamp(mtime)))
@staticmethod
def inode(path):
return str(System.inode(path))
return System.inode(path)
def _do_update(self, path, dump=True):
def _do_update(self, path):
if not os.path.exists(path):
return (None, None)
mtime = self.mtime(path)
inode = self.inode(path)
md5 = self._get(inode, mtime)
if md5:
return (md5, None)
cmd = 'SELECT * from {} WHERE inode={}'.format(self.STATE_TABLE,
inode)
md5, info = self._collect(path)
state = StateEntry(md5, mtime)
d = state.dumpd()
with self._lock:
with self._lock_file:
self._db[inode] = d
if dump:
self.dump()
self._execute(cmd)
ret = self.c.fetchall()
if len(ret) == 0:
md5, info = self._collect(path)
cmd = 'INSERT INTO {}(inode, mtime, md5, timestamp) ' \
'VALUES ({}, "{}", "{}", "{}")'
self._execute(cmd.format(self.STATE_TABLE,
inode,
mtime,
md5,
int(nanotime.timestamp(time.time()))))
self.inserts += 1
else:
assert len(ret) == 1
assert len(ret[0]) == 4
i, m, md5, timestamp = ret[0]
assert i == inode
if mtime != m:
md5, info = self._collect(path)
cmd = 'UPDATE {} SET ' \
'mtime = "{}", md5 = "{}", timestamp = "{}" ' \
'WHERE inode = {}'
self._execute(cmd.format(self.STATE_TABLE,
mtime,
md5,
int(nanotime.timestamp(time.time())),
inode))
else:
info = None
cmd = 'UPDATE {} SET timestamp = "{}" WHERE inode = {}'
self._execute(cmd.format(self.STATE_TABLE,
int(nanotime.timestamp(time.time())),
inode))
return (md5, info)
def update(self, path, dump=True):
return self._do_update(path, dump=dump)[0]
def update(self, path):
return self._do_update(path)[0]
def update_info(self, path, dump=True):
md5, info = self._do_update(path, dump=dump)
def update_info(self, path):
md5, info = self._do_update(path)
if not info:
info = self.project.cache.local.load_dir_cache(md5)
return (md5, info)
def _get(self, inode, mtime):
with self._lock:
with self._lock_file:
d = self._db.get(inode, None)
if not d:
return None
state = StateEntry.loadd(d)
if mtime == state.mtime:
return state.md5
return None
def get(self, path):
mtime = self.mtime(path)
inode = self.inode(path)
return self._get(inode, mtime)
class LinkStateEntry(object):
PARAM_MTIME = 'mtime'
PARAM_INODE = 'inode'
def __init__(self, inode, mtime):
self.mtime = mtime
self.inode = inode
@staticmethod
def loadd(d):
mtime = d[LinkStateEntry.PARAM_MTIME]
inode = d[LinkStateEntry.PARAM_INODE]
return LinkStateEntry(inode, mtime)
def dumpd(self):
return {
self.PARAM_INODE: self.inode,
self.PARAM_MTIME: self.mtime,
}
class LinkState(State):
STATE_FILE = 'link.state'
STATE_LOCK_FILE = STATE_FILE + '.lock'
def __init__(self, project):
super(LinkState, self).__init__(project)
self.root_dir = project.root_dir
def update(self, path, dump=True):
def update_link(self, path):
if not os.path.exists(path):
return
mtime = self.mtime(path)
inode = self.inode(path)
relpath = os.path.relpath(path, self.root_dir)
with self._lock:
state = LinkStateEntry(inode, mtime)
d = state.dumpd()
self._db[os.path.relpath(path, self.root_dir)] = d
if dump:
with self._lock_file:
self.dump()
cmd = 'REPLACE INTO {}(path, inode, mtime) ' \
'VALUES ("{}", {}, "{}")'.format(self.LINK_STATE_TABLE,
relpath,
inode,
mtime)
self._execute(cmd)
def _do_remove_unused(self, used):
items = self._db.copy().items()
for p, s in items:
def remove_unused_links(self, used):
unused = []
self._execute('SELECT * FROM {}'.format(self.LINK_STATE_TABLE))
for row in self.c:
p, i, m = row
path = os.path.join(self.root_dir, p)
state = LinkStateEntry.loadd(s)
if path in used:
continue
@@ -240,13 +306,11 @@ class LinkState(State):
inode = self.inode(path)
mtime = self.mtime(path)
if inode == state.inode and mtime == state.mtime:
if i == inode and m == mtime:
Logger.debug('Removing \'{}\' as unused link.'.format(path))
remove(path)
del self._db[p]
unused.append(p)
def remove_unused(self, used):
with self._lock:
with self._lock_file:
self._do_remove_unused(used)
self.dump()
for p in unused:
cmd = 'DELETE FROM {} WHERE path = "{}"'
self._execute(cmd.format(self.LINK_STATE_TABLE, p))
+138 -19
View File
@@ -1,10 +1,4 @@
import os
import ctypes
import reflink
if os.name == 'nt':
import ntfsutils.hardlink as winlink
from ntfsutils.fs import getdirinfo
class System(object):
@@ -14,13 +8,24 @@ class System(object):
@staticmethod
def hardlink(source, link_name):
import ctypes
if System.is_unix():
return os.link(source, link_name)
os.link(source, link_name)
return
return winlink.create(source, link_name)
CreateHardLink = ctypes.windll.kernel32.CreateHardLinkW
CreateHardLink.argtypes = [ctypes.c_wchar_p,
ctypes.c_wchar_p,
ctypes.c_void_p]
CreateHardLink.restype = ctypes.wintypes.BOOL
res = CreateHardLink(link_name, source, None)
if res == 0:
raise ctypes.WinError()
@staticmethod
def symlink(source, link_name):
import ctypes
if System.is_unix():
return os.symlink(source, link_name)
@@ -36,26 +41,138 @@ class System(object):
raise ctypes.WinError()
@staticmethod
def _reflink_darwin(src, dst):
import ctypes
clib = ctypes.CDLL('libc.dylib')
if not hasattr(clib, 'clonefile'):
return -1
clonefile = clib.clonefile
clonefile.argtypes = [ctypes.c_char_p,
ctypes.c_char_p,
ctypes.c_int]
clonefile.restype = ctypes.c_int
return clonefile(ctypes.c_char_p(src.encode('utf-8')),
ctypes.c_char_p(dst.encode('utf-8')),
ctypes.c_int(0))
@staticmethod
def _reflink_windows(src, dst):
return -1
@staticmethod
def _reflink_linux(src, dst):
import os
import fcntl
FICLONE = 0x40049409
s = open(src, 'r')
d = open(dst, 'w+')
try:
ret = fcntl.ioctl(d.fileno(), FICLONE, s.fileno())
except IOError:
s.close()
d.close()
os.unlink(dst)
raise
s.close()
d.close()
if ret != 0:
os.unlink(dst)
return ret
@staticmethod
def reflink(source, link_name):
return reflink.reflink(source, link_name)
import platform
from dvc.exceptions import DvcException
system = platform.system()
try:
if system == 'Windows':
ret = System._reflink_windows(source, link_name)
elif system == 'Darwin':
ret = System._reflink_darwin(source, link_name)
elif system == 'Linux':
ret = System._reflink_linux(source, link_name)
else:
ret = -1
except IOError:
ret = -1
if ret != 0:
raise DvcException('Reflink is not supported')
# FIXME
# Temporary fix while waiting for the PR to be merged and released:
# https://github.com/sunshowers/ntfs/pull/11
@staticmethod
def getdirinfo(path):
from ntfsutils.fs import FILE_FLAG_BACKUP_SEMANTICS, FILE_SHARE_READ, OPEN_EXISTING
from ntfsutils.fs import CreateFile, WinError, BY_HANDLE_FILE_INFORMATION
from ntfsutils.fs import GetFileInformationByHandle, CloseHandle
import ctypes
from ctypes import c_void_p, c_wchar_p, Structure, WinError
from ctypes.wintypes import DWORD, HANDLE, POINTER, BOOL
FILE_FLAG_BACKUP_SEMANTICS = 0x02000000
FILE_SHARE_READ = 0x00000001
OPEN_EXISTING = 3
class FILETIME(Structure):
_fields_ = [("dwLowDateTime", DWORD),
("dwHighDateTime", DWORD)]
class BY_HANDLE_FILE_INFORMATION(Structure):
_fields_ = [("dwFileAttributes", DWORD),
("ftCreationTime", FILETIME),
("ftLastAccessTime", FILETIME),
("ftLastWriteTime", FILETIME),
("dwVolumeSerialNumber", DWORD),
("nFileSizeHigh", DWORD),
("nFileSizeLow", DWORD),
("nNumberOfLinks", DWORD),
("nFileIndexHigh", DWORD),
("nFileIndexLow", DWORD)]
flags = FILE_FLAG_BACKUP_SEMANTICS
hfile = CreateFile(path, 0, FILE_SHARE_READ, None, OPEN_EXISTING, flags, None)
func = ctypes.windll.kernel32.CreateFileW
func.argtypes = [c_wchar_p,
DWORD,
DWORD,
c_void_p,
DWORD,
DWORD,
HANDLE]
func.restype = HANDLE
hfile = func(path,
0,
FILE_SHARE_READ,
None,
OPEN_EXISTING,
flags,
None)
if hfile is None:
raise WinError()
func = ctypes.windll.kernel32.GetFileInformationByHandle
func.argtypes = [HANDLE, POINTER(BY_HANDLE_FILE_INFORMATION)]
func.restype = BOOL
info = BY_HANDLE_FILE_INFORMATION()
rv = GetFileInformationByHandle(hfile, info)
CloseHandle(hfile)
rv = func(hfile, info)
func = ctypes.windll.kernel32.CloseHandle
func.argtypes = [HANDLE]
func.restype = BOOL
func(hfile)
if rv == 0:
raise WinError()
return info
@staticmethod
@@ -65,4 +182,6 @@ class System(object):
# getdirinfo from ntfsutils works on both files and dirs
info = System.getdirinfo(path)
return hash((info.dwVolumeSerialNumber, info.nFileIndexHigh, info.nFileIndexLow))
return hash((info.dwVolumeSerialNumber,
info.nFileIndexHigh,
info.nFileIndexLow))
+14 -7
View File
@@ -5,10 +5,13 @@ import requests
from dvc import VERSION_BASE
from dvc.logger import Logger
class Updater(object):
URL = 'https://4ki8820rsf.execute-api.us-east-2.amazonaws.com/prod/latest-version'
class Updater(object): # pragma: no cover
URL = 'https://4ki8820rsf.execute-api.us-east-2.amazonaws.com/' \
'prod/latest-version'
UPDATER_FILE = 'updater'
TIMEOUT = 7 * 24 * 60 * 60 #every week
TIMEOUT = 24 * 60 * 60 # every day
TIMEOUT_GET = 10
def __init__(self, dvc_dir):
self.dvc_dir = dvc_dir
@@ -33,13 +36,16 @@ class Updater(object):
os.unlink(self.updater_file)
Logger.info('Checking for updates...')
try:
r = requests.get(self.URL)
r = requests.get(self.URL, timeout=self.TIMEOUT_GET)
j = r.json()
latest = j['version']
open(self.updater_file, 'w+').close()
except Exception as exc:
Logger.debug('Failed to obtain latest version: {}'.format(str(exc)))
msg = 'Failed to obtain latest version: {}'.format(str(exc))
Logger.debug(msg)
return
l_major, l_minor, l_patch = [int(x) for x in latest.split('.')]
@@ -48,7 +54,8 @@ class Updater(object):
if l_major <= c_major and \
l_minor <= c_minor and \
l_patch <= c_patch:
return
return
msg = 'You are using dvc version {}, however version {} is available. Consider upgrading.'
msg = 'You are using dvc version {}, however version {} is ' \
'available. Consider upgrading.'
Logger.warn(msg.format(current, latest))
+40 -5
View File
@@ -2,21 +2,20 @@
Helpers for other modules.
"""
import os
import re
import sys
import math
import json
import stat
import shutil
import hashlib
from multiprocessing.pool import ThreadPool
from dvc.istextfile import istextfile
from dvc.progress import progress
from dvc.istextfile import istextfile
from dvc.logger import Logger
LOCAL_CHUNK_SIZE = 1024*1024
LARGE_FILE_SIZE = 1024*1024*1024
LARGE_DIR_SIZE = 100
def dos2unix(data):
@@ -28,6 +27,14 @@ def file_md5(fname):
if os.path.exists(fname):
hash_md5 = hashlib.md5()
binary = not istextfile(fname)
size = os.path.getsize(fname)
bar = False
if size >= LARGE_FILE_SIZE:
bar = True
msg = "Computing md5 for a large file {}. This is only done once."
Logger.info(msg.format(os.path.relpath(fname)))
name = os.path.relpath(fname)
total = 0
with open(fname, 'rb') as fobj:
while True:
@@ -35,6 +42,10 @@ def file_md5(fname):
if not data:
break
if bar:
total += len(data)
progress.update_target(name, total, size)
if binary:
chunk = data
else:
@@ -42,6 +53,9 @@ def file_md5(fname):
hash_md5.update(chunk)
if bar:
progress.finish_target(name)
return (hash_md5.hexdigest(), hash_md5.digest())
else:
return (None, None)
@@ -67,7 +81,7 @@ def copyfile(src, dest, no_progress_bar=False, name=None):
fsrc = open(src, 'rb')
if os.path.isdir(dest):
fdest = open(dest + '/' + os.path.basename(src), 'wb+')
fdest = open(os.path.join(dest, os.path.basename(src)), 'wb+')
else:
fdest = open(dest, 'wb+')
@@ -121,3 +135,24 @@ def to_chunks(l, jobs):
n = 1
return [l[x:x+n] for x in range(0, len(l), n)]
# NOTE: Fix env variables modified by PyInstaller
# http://pyinstaller.readthedocs.io/en/stable/runtime-information.html
def fix_env(env):
if env is None:
env = os.environ.copy()
else:
env = env.copy()
# NOTE: Check if we are in a bundle
# https://pythonhosted.org/PyInstaller/runtime-information.html
if getattr(sys, 'frozen', False):
lp_key = 'LD_LIBRARY_PATH'
lp_orig = env.get(lp_key + '_ORIG', None)
if lp_orig is not None:
env[lp_key] = lp_orig
else:
env.pop(lp_key, None)
return env
+8 -6
View File
@@ -1,23 +1,25 @@
azure-storage-blob==1.3.0
boto3==1.7.4
ply==3.8 # required by google-gax
ply>=3.9 # See https://github.com/pyinstaller/pyinstaller/issues/1945
configparser>=3.5.0
zc.lockfile>=1.2.1
future>=0.16.0
google-cloud==0.32.0
google-cloud-storage==1.12.0
PyInstaller==3.3.1
colorama>=0.3.9
configobj>=5.0.6
networkx>=1.11
networkx>=2.1
pyyaml>=3.12
gitpython>=2.1.8
ntfsutils>=0.1.4
setuptools>=34.0.0
pytest>=3.3.1
nanotime>=0.5.2
pyasn1>=0.4.1
schema>=0.6.7
paramiko>=2.4.1
jsonpath-rw==1.4.0
reflink==0.2.0
requests>=2.18.4
wheel>=0.31.1
futures>=3.2.0; python_version == "2.7"
grandalf==0.6
pydot>=1.2.4
asciimatics>=1.10.0
+7
View File
@@ -17,3 +17,10 @@ ssh-keyscan 0.0.0.0 >> ~/.ssh/known_hosts
ssh 0.0.0.0 ls &> /dev/null
ssh 127.0.0.1 ls &> /dev/null
ssh localhost ls &> /dev/null
scriptdir="$(dirname $0)"
if [ -n "$TRAVIS_OS_NAME" ] && [ "$TRAVIS_OS_NAME" != "osx" ]; then
bash "$scriptdir/install_azurite.sh"
source ~/.bashrc
fi
+15 -3
View File
@@ -3,9 +3,21 @@
set -x
set -e
pip install --upgrade pip
pip install -r requirements.txt
pip install -r test-requirements.txt
N_RETRIES=3
function retry {
for i in $(seq $N_RETRIES); do
$@ && break
done
}
# NOTE: it is not uncommon for pip to hang on travis for what seems to be
# networking issues. Thus, let's retry a few times to see if it will eventially
# work or not.
retry pip install --upgrade pip setuptools wheel
retry pip install -r requirements.txt
retry pip install -r test-requirements.txt
git config --global user.email "[email protected]"
git config --global user.name "DVC Tester"
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
set -euo pipefail
# install docker
export DEBIAN_FRONTEND=noninteractive
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y docker-ce
# run azurite
sudo docker run --restart always -e executable=blob -p 10000:10000 --tmpfs /opt/azurite/folder -d arafato/azurite:2.6.5
# save secrets
echo "export AZURE_STORAGE_CONTAINER_NAME='travis-tests'" >> ~/.bashrc
echo "export AZURE_STORAGE_CONNECTION_STRING='DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;'" >> ~/.bashrc
+1 -11
View File
@@ -5,18 +5,8 @@
set -x
set -e
# This command is used to tell shell, turn installation mode to non interactive
# and set auto selection of agreement for Sun Java
export DEBIAN_FRONTEND=noninteractive
echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections
echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections
sudo apt-get update -y
sudo apt-get install python-software-properties -y
sudo add-apt-repository ppa:webupd8team/java -y
sudo apt-get update -y
sudo -E apt-get install oracle-java8-installer -y
sudo apt-get install default-jdk
java -version
pushd /usr/local
+24 -2
View File
@@ -6,11 +6,33 @@ set -e
python -mtests
if [[ "$TRAVIS_PULL_REQUEST" == "false" && \
"$TRAVIS_SECURE_ENV_VARS" == "true" ]]; then
"$TRAVIS_SECURE_ENV_VARS" == "true" && \
"$(python -c 'import sys; print(sys.version_info[0])')" == '2' ]]; then
codecov
fi
if [[ ! -z "$TRAVIS_TAG" ]]; then
./scripts/build_package.sh
./scripts/build_posix.sh
# Test version
pip uninstall -y dvc
if [ -x "$(command -v dvc)" ]; then
echo "ERROR: dvc command already exists!!!"
exit 1
fi
pip install dist/dvc-*.whl
if [[ "$(dvc --version)" != "$TRAVIS_TAG" ]]; then
echo "ERROR: 'dvc --version'$(dvc -V) doesn't match '$TRAVIS_TAG'"
exit 1
fi
pip uninstall -y dvc
PY_VER=$(python -c 'import sys; print(sys.version_info[0:2])')
if [[ "$PY_VER" != '(3, 7)' ]]; then
./scripts/build_posix.sh
else
echo "Not building dvc binary package on Python 3.7, since it is "\
"not supported by PyInstaller yet."
fi
fi
+15 -9
View File
@@ -1,3 +1,4 @@
import sys
import platform
from setuptools import setup, find_packages
from distutils.errors import DistutilsPlatformError
@@ -5,37 +6,39 @@ from dvc import VERSION
install_requires = [
"ply==3.8", # required by google-gax
"ply>=3.9", # See https://github.com/pyinstaller/pyinstaller/issues/1945
"configparser>=3.5.0",
"zc.lockfile>=1.2.1",
"future>=0.16.0",
"colorama>=0.3.9",
"configobj>=5.0.6",
"networkx>=1.11",
"networkx>=2.1",
"pyyaml>=3.12",
"gitpython>=2.1.8",
"ntfsutils>=0.1.4",
"setuptools>=34.0.0",
"pytest>=3.3.1",
"nanotime>=0.5.2",
"pyasn1>=0.4.1",
"schema>=0.6.7",
"jsonpath-rw==1.4.0",
"reflink==0.2.0",
"requests>=2.18.4",
"grandalf==0.6",
"asciimatics>=1.10.0",
]
# Extra dependencies for remote integrations
gs = [
"google-cloud==0.32.0",
"google-cloud-storage==1.12.0",
]
s3 = [
"boto3==1.7.4",
]
azure = [
"azure-storage-blob==1.3.0"
]
ssh = [
"paramiko>=2.4.1",
]
all_remotes = gs + s3 + ssh
all_remotes = gs + s3 + azure + ssh
setup(
name='dvc',
@@ -51,7 +54,10 @@ setup(
'all': all_remotes,
'gs': gs,
's3': s3,
'azure': azure,
'ssh': ssh,
# NOTE: https://github.com/inveniosoftware/troubleshooting/issues/1
':python_version=="2.7"': ['futures'],
},
keywords='data science, data version control, machine learning',
classifiers=[
@@ -59,11 +65,11 @@ setup(
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 3',
],
packages=find_packages(exclude=['bin', 'tests', 'functests']),
packages=find_packages(exclude=['tests']),
include_package_data=True,
url='http://dataversioncontrol.com',
entry_points={
'console_scripts': ['dvc = dvc.main:main']
},
zip_safe=False
zip_safe=False,
)
+5 -2
View File
@@ -1,8 +1,11 @@
pytest>=3.3.1
nose>=1.3.7
mock>=2.0.0
coverage
codecov
coverage>=4.5.1
codecov>=2.0.15
xmltodict>=0.11.0
awscli>=1.14.18
google-compute-engine
pywin32; sys_platform == 'win32'
Pygments
collective.checkdocs
+2 -1
View File
@@ -7,4 +7,5 @@ os.chdir(REPO_ROOT)
os.putenv('PATH', '{}:{}'.format(os.path.join(REPO_ROOT, 'bin'), os.getenv('PATH')))
os.putenv('DVC_HOME', REPO_ROOT)
check_call('nosetests --processes=-1 --process-timeout=200 --cover-inclusive --cover-erase --cover-package=dvc --with-coverage', shell=True)
cmd = 'nosetests -v --processes=-1 --process-timeout=200 --cover-inclusive --cover-erase --cover-package=dvc --with-coverage'
check_call(cmd, shell=True)
+9
View File
@@ -61,6 +61,15 @@ class TestGit(TestDir):
self.git.index.commit('add code')
class TestGitSubmodule(TestGit):
def setUp(self):
super(TestGitSubmodule, self).setUp()
subrepo = Repo.init()
subrepo_path = 'subrepo'
self.git.create_submodule(subrepo_path, subrepo_path, subrepo.git_dir)
self._pushd(subrepo_path)
class TestDvc(TestGit):
def setUp(self):
super(TestDvc, self).setUp()
-1
View File
@@ -1 +0,0 @@
/home/efiop/virtenvs/python3-dvc/bin/python: No module named tests.__main__; 'tests' is a package and cannot be directly executed
+81 -10
View File
@@ -2,10 +2,12 @@ import os
import stat
import shutil
import filecmp
import tempfile
from dvc.main import main
from dvc.utils import file_md5
from dvc.stage import Stage
from dvc.exceptions import DvcException
from dvc.output.base import OutputAlreadyTrackedError
from dvc.output.base import OutputDoesNotExistError, OutputIsNotFileOrDirError
from dvc.command.add import CmdAdd
@@ -17,7 +19,10 @@ class TestAdd(TestDvc):
def test(self):
md5 = file_md5(self.FOO)[0]
stage = self.dvc.add(self.FOO)
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertTrue(stage is not None)
self.assertIsInstance(stage, Stage)
self.assertTrue(os.path.isfile(stage.path))
@@ -27,10 +32,10 @@ class TestAdd(TestDvc):
self.assertEqual(stage.outs[0].info['md5'], md5)
class TestAddNonExistentFile(TestDvc):
class TestAddUnupportedFile(TestDvc):
def test(self):
with self.assertRaises(OutputDoesNotExistError) as cx:
self.dvc.add('non_existent_file')
with self.assertRaises(DvcException) as cx:
self.dvc.add('unsupported://unsupported')
class TestAddDirectory(TestDvc):
@@ -38,14 +43,43 @@ class TestAddDirectory(TestDvc):
dname = 'directory'
os.mkdir(dname)
self.create(os.path.join(dname, 'file'), 'file')
self.dvc.add(dname)
stages = self.dvc.add(dname)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertTrue(stage is not None)
self.assertEqual(len(stage.deps), 0)
self.assertEqual(len(stage.outs), 1)
md5 = stage.outs[0].info['md5']
dir_info = self.dvc.cache.local.load_dir_cache(md5)
for info in dir_info:
self.assertTrue('\\' not in info['relpath'])
class TestAddDirectoryRecursive(TestDvc):
def test(self):
stages = self.dvc.add(self.DATA_DIR, recursive=True)
self.assertEqual(len(stages), 2)
class TestAddCmdDirectoryRecursive(TestDvc):
def test(self):
ret = main(['add',
'--recursive',
self.DATA_DIR])
self.assertEqual(ret, 0)
class TestAddDirectoryWithForwardSlash(TestDvc):
def test(self):
dname = 'directory/'
os.mkdir(dname)
self.create(os.path.join(dname, 'file'), 'file')
stage = self.dvc.add(dname)
stages = self.dvc.add(dname)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertTrue(stage is not None)
self.assertEquals(os.path.abspath('directory.dvc'), stage.path)
class TestAddTrackedFile(TestDvc):
@@ -66,15 +100,52 @@ class TestAddDirWithExistingCache(TestDvc):
os.mkdir(dname)
shutil.copyfile(self.FOO, fname)
self.dvc.add(self.FOO)
self.dvc.add(dname)
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
stages = self.dvc.add(dname)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
class TestAddModifiedDir(TestDvc):
def test(self):
self.dvc.add(self.DATA_DIR)
stages = self.dvc.add(self.DATA_DIR)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
os.unlink(self.DATA)
self.dvc.add(self.DATA_DIR)
stages = self.dvc.add(self.DATA_DIR)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
class TestAddFileInDir(TestDvc):
def test(self):
stages = self.dvc.add(self.DATA_SUB)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertNotEqual(stage, None)
self.assertEqual(len(stage.deps), 0)
self.assertEqual(len(stage.outs), 1)
self.assertEqual(stage.relpath, self.DATA_SUB + '.dvc')
class TestAddExternalLocalFile(TestDvc):
def test(self):
dname = tempfile.mkdtemp()
fname = os.path.join(dname, 'foo')
shutil.copyfile(self.FOO, fname)
stages = self.dvc.add(fname)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertNotEqual(stage, None)
self.assertEqual(len(stage.deps), 0)
self.assertEqual(len(stage.outs), 1)
self.assertEqual(stage.relpath, 'foo.dvc')
self.assertEqual(len(os.listdir(dname)), 1)
self.assertTrue(os.path.isfile(fname))
self.assertTrue(filecmp.cmp(fname, 'foo', shallow=False))
class TestCmdAdd(TestDvc):
+9
View File
@@ -103,3 +103,12 @@ class TestSharedCacheDir(TestDir):
self.assertEqual(len(os.listdir(os.path.join(cache_dir, subdirs[0]))), 1)
self.assertEqual(len(os.listdir(os.path.join(cache_dir, subdirs[1]))), 1)
self.assertEqual(len(os.listdir(os.path.join(cache_dir, subdirs[2]))), 1)
class TestCacheLinkType(TestDvc):
def test(self):
ret = main(['config', 'cache.type', 'reflink,copy'])
self.assertEqual(ret, 0)
ret = main(['add', self.FOO])
self.assertEqual(ret, 0)
+35 -9
View File
@@ -16,7 +16,11 @@ class TestCheckout(TestRepro):
def setUp(self):
super(TestCheckout, self).setUp()
self.data_dir_stage = self.dvc.add(self.DATA_DIR)
stages = self.dvc.add(self.DATA_DIR)
self.assertEqual(len(stages), 1)
self.data_dir_stage = stages[0]
self.assertTrue(self.data_dir_stage is not None)
self.orig = 'orig'
shutil.copy(self.FOO, self.orig)
@@ -32,7 +36,7 @@ class TestCheckout(TestRepro):
def _test_checkout(self):
self.assertTrue(os.path.isfile(self.FOO))
self.assertTrue(filecmp.cmp(self.FOO, self.orig))
self.assertTrue(filecmp.cmp(self.FOO, self.orig, shallow=False))
class TestCheckoutSingleStage(TestCheckout):
@@ -74,7 +78,9 @@ class CheckoutBase(TestDvc):
def commit_data_file(self, fname, content='random text'):
with open(fname, 'w') as fd:
fd.write(content)
self.dvc.add(fname)
stages = self.dvc.add(fname)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
self.dvc.scm.add([fname + '.dvc', '.gitignore'])
self.dvc.scm.commit('adding ' + fname)
@@ -93,7 +99,8 @@ class TestRemoveFilesWhenCheckout(CheckoutBase):
# add the file into a separate branch
self.dvc.scm.checkout(branch_1, True)
main(['checkout'])
ret = main(['checkout'])
self.assertEqual(ret, 0)
self.commit_data_file(fname)
# Checkout back in master
@@ -102,7 +109,8 @@ class TestRemoveFilesWhenCheckout(CheckoutBase):
# Make sure `dvc checkout` removes the file
# self.dvc.checkout()
main(['checkout'])
ret = main(['checkout'])
self.assertEqual(ret, 0)
self.assertFalse(os.path.exists(fname))
@@ -145,11 +153,13 @@ class TestGitIgnoreWhenCheckout(CheckoutBase):
self.commit_data_file(fname_master)
self.dvc.scm.checkout(branch_1, True)
main(['checkout'])
ret = main(['checkout'])
self.assertEqual(ret, 0)
self.commit_data_file(fname_branch)
self.dvc.scm.checkout(branch_master)
main(['checkout'])
ret = main(['checkout'])
self.assertEqual(ret, 0)
ignored = self.read_ignored()
@@ -157,7 +167,8 @@ class TestGitIgnoreWhenCheckout(CheckoutBase):
self.assertIn(fname_master, ignored)
self.dvc.scm.checkout(branch_1)
main(['checkout'])
ret = main(['checkout'])
self.assertEqual(ret, 0)
ignored = self.read_ignored()
self.assertIn(fname_branch, ignored)
@@ -181,7 +192,10 @@ class TestCheckoutEmptyDir(TestDvc):
dname = 'empty_dir'
os.mkdir(dname)
stage = self.dvc.add(dname)
stages = self.dvc.add(dname)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertTrue(stage is not None)
self.assertEqual(len(stage.outs), 1)
stage.outs[0].remove()
@@ -191,3 +205,15 @@ class TestCheckoutEmptyDir(TestDvc):
self.assertTrue(os.path.isdir(dname))
self.assertEqual(len(os.listdir(dname)), 0)
class TestCheckoutNotCachedFile(TestDvc):
def test(self):
cmd = 'python {} {} {}'.format(self.CODE, self.FOO, 'out')
self.dvc.add(self.FOO)
stage = self.dvc.run(cmd=cmd,
deps=[self.FOO, self.CODE],
outs_no_cache=['out'])
self.dvc.checkout()
+30
View File
@@ -1,6 +1,7 @@
import os
from dvc.cli import parse_args
from dvc.main import main
from dvc.command.init import CmdInit
from dvc.command.run import CmdRun
from dvc.command.data_sync import CmdDataPull, CmdDataPush
@@ -11,6 +12,8 @@ from dvc.command.remove import CmdRemove
from dvc.command.gc import CmdGC
from dvc.command.config import CmdConfig
from dvc.command.checkout import CmdCheckout
from dvc.exceptions import DvcException
from dvc.command.base import CmdBase
from tests.basic_env import TestDvc
@@ -146,3 +149,30 @@ class TestCheckout(TestDvc):
def test(self):
args = parse_args(['checkout'])
self.assertIsInstance(args.func(args), CmdCheckout)
class TestStartupTime(TestDvc):
def test(self):
import timeit
start = timeit.default_timer()
try:
main([])
except SystemExit:
pass
t = timeit.default_timer() - start
self.assertTrue(t < 0.3)
class TestFindRoot(TestDvc):
def test(self):
os.chdir("..")
class A(object):
quiet = False
verbose = True
args = A()
with self.assertRaises(DvcException):
CmdBase(args)
+48 -12
View File
@@ -1,12 +1,18 @@
import configobj
from dvc.main import main
from dvc.command.config import CmdConfig
from tests.basic_env import TestDvc
class TestConfigCLI(TestDvc):
def _contains(self, section, field, value):
config = configobj.ConfigObj(self.dvc.config.config_file)
def _contains(self, section, field, value, local=False):
if local:
fname = self.dvc.config.config_local_file
else:
fname = self.dvc.config.config_file
config = configobj.ConfigObj(fname)
if section not in config.keys():
return False
@@ -22,28 +28,38 @@ class TestConfigCLI(TestDvc):
ret = main(['root'])
self.assertEqual(ret, 0)
def test(self):
def _do_test(self, local=False):
section = 'setsection'
field = 'setfield'
section_field = '{}.{}'.format(section, field)
value = 'setvalue'
newvalue = 'setnewvalue'
ret = main(['config', section_field, value])
self.assertEqual(ret, 0)
self.assertTrue(self._contains(section, field, value))
base = ['config']
if local:
base.append('--local')
ret = main(['config', section_field])
ret = main(base + [section_field, value])
self.assertEqual(ret, 0)
self.assertTrue(self._contains(section, field, value, local))
ret = main(base + [section_field])
self.assertEqual(ret, 0)
ret = main(['config', section_field, newvalue])
ret = main(base + [section_field, newvalue])
self.assertEqual(ret, 0)
self.assertTrue(self._contains(section, field, newvalue))
self.assertFalse(self._contains(section, field, value))
self.assertTrue(self._contains(section, field, newvalue, local))
self.assertFalse(self._contains(section, field, value, local))
ret = main(['config', section_field, '--unset'])
ret = main(base + [section_field, '--unset'])
self.assertEqual(ret, 0)
self.assertFalse(self._contains(section, field, value))
self.assertFalse(self._contains(section, field, value, local))
def test(self):
self._do_test(False)
def test_local(self):
self._do_test(True)
def test_non_existing(self):
#FIXME check stdout/err
@@ -59,3 +75,23 @@ class TestConfigCLI(TestDvc):
ret = main(['config', 'global.non_existing_field', '-u'])
self.assertEqual(ret, 1)
ret = main(['config', 'core.remote', 'myremote'])
self.assertEqual(ret, 0)
ret = main(['config', 'core.non_existing_field', '-u'])
self.assertEqual(ret, 1)
def test_failed_write(self):
class A(object):
local = False
name = 'core.remote'
value = 'myremote'
unset = False
args = A()
cmd = CmdConfig(args)
cmd.configobj.write = None
ret = cmd.save()
self.assertNotEqual(ret, 0)
+170 -75
View File
@@ -1,6 +1,8 @@
from subprocess import CalledProcessError
from subprocess import check_output, Popen
from unittest import SkipTest
import os
import time
import stat
import uuid
import shutil
import getpass
@@ -9,7 +11,8 @@ import platform
from dvc.main import main
from dvc.config import Config, ConfigError
from dvc.data_cloud import DataCloud, RemoteS3, RemoteGS, RemoteLOCAL, RemoteSSH, RemoteHDFS
from dvc.data_cloud import (DataCloud, RemoteS3, RemoteGS, RemoteAzure,
RemoteLOCAL, RemoteSSH, RemoteHDFS)
from dvc.remote.base import STATUS_OK, STATUS_NEW, STATUS_DELETED
from tests.basic_env import TestDvc
@@ -46,28 +49,59 @@ def _should_test_gcp():
if os.path.exists(creds):
os.unlink(creds)
shutil.copyfile(TestDvc.GCP_CREDS_FILE, creds)
ret = os.system('gcloud auth activate-service-account --key-file={}'.format(creds))
assert ret == 0
try:
check_output(['gcloud', 'auth', 'activate-service-account',
'--key-file', creds])
except (CalledProcessError, OSError):
return False
return True
return False
def _should_test_azure():
if os.getenv("DVC_TEST_AZURE") == "true":
return True
elif os.getenv("DVC_TEST_AZURE") == "false":
return False
return (os.getenv("AZURE_STORAGE_CONTAINER_NAME")
and os.getenv("AZURE_STORAGE_CONNECTION_STRING"))
def _should_test_ssh():
if os.getenv("DVC_TEST_SSH") == "true":
return True
#FIXME: enable on windows
# FIXME: enable on windows
if os.name == 'nt':
return False
assert os.system('ssh 127.0.0.1 ls &> /dev/null') == 0
try:
check_output(['ssh', '-o', 'BatchMode=yes', '127.0.0.1', 'ls'])
except (CalledProcessError, IOError):
return False
return True
def _should_test_hdfs():
return platform.system() == 'Linux'
if platform.system() != 'Linux':
return False
try:
check_output(['hadoop', 'version'], shell=True, executable=os.getenv('SHELL'))
except (CalledProcessError, IOError):
return False
p = Popen('hadoop fs -ls hdfs://127.0.0.1/',
shell=True,
executable=os.getenv('SHELL'))
p.communicate()
if p.returncode != 0:
return False
return True
def get_local_storagepath():
@@ -79,11 +113,15 @@ def get_local_url():
def get_ssh_url():
return 'ssh://{}@127.0.0.1:{}'.format(getpass.getuser(), get_local_storagepath())
return 'ssh://{}@127.0.0.1:{}'.format(
getpass.getuser(),
get_local_storagepath())
def get_hdfs_url():
return 'hdfs://{}@127.0.0.1{}'.format(getpass.getuser(), get_local_storagepath())
return 'hdfs://{}@127.0.0.1{}'.format(
getpass.getuser(),
get_local_storagepath())
def get_aws_storagepath():
@@ -102,6 +140,10 @@ def get_gcp_url():
return 'gs://' + get_gcp_storagepath()
def get_azure_url():
return 'azure://'
def sleep():
# Sleep some time to simulate realistic behavior.
# Some filesystems have a bad date resolution for
@@ -119,30 +161,44 @@ class TestDataCloud(TestDvc):
def test(self):
config = TEST_CONFIG
for scheme, cl in [('s3://', RemoteS3),
('gs://', RemoteGS),
('ssh://user@localhost:/', RemoteSSH),
(tempfile.mkdtemp(), RemoteLOCAL)]:
config[TEST_SECTION][Config.SECTION_REMOTE_URL] = scheme + str(uuid.uuid4())
clist = [('s3://', RemoteS3),
('gs://', RemoteGS),
('ssh://user@localhost:/', RemoteSSH),
(tempfile.mkdtemp(), RemoteLOCAL)]
if _should_test_azure():
clist.append(('azure://ContainerName=', RemoteAzure))
for scheme, cl in clist:
remote_url = scheme + str(uuid.uuid4())
config[TEST_SECTION][Config.SECTION_REMOTE_URL] = remote_url
self._test_cloud(config, cl)
def test_unsupported(self):
with self.assertRaises(ConfigError) as cx:
with self.assertRaises(ConfigError):
remote_url = 'notsupportedscheme://a/b'
config = TEST_CONFIG
config[TEST_SECTION][Config.SECTION_REMOTE_URL] = 'notsupportedscheme://a/b'
config[TEST_SECTION][Config.SECTION_REMOTE_URL] = remote_url
DataCloud(self.dvc, config=config)
class TestDataCloudBase(TestDvc):
def _get_cloud_class(self):
return None
def _should_test(self):
return False
def _get_url(self):
return None
return ''
def _ensure_should_run(self):
if not self._should_test():
raise SkipTest('Test {} is disabled'
.format(self.__class__.__name__))
def _setup_cloud(self):
if not self._should_test():
return
self._ensure_should_run()
repo = self._get_url()
@@ -155,69 +211,77 @@ class TestDataCloudBase(TestDvc):
def _test_cloud(self):
self._setup_cloud()
stage = self.dvc.add(self.FOO)
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertTrue(stage is not None)
cache = stage.outs[0].cache
info = stage.outs[0].info
info = stage.outs[0].dumpd()
md5 = info['md5']
stage_dir = self.dvc.add(self.DATA_DIR)
stages = self.dvc.add(self.DATA_DIR)
self.assertEqual(len(stages), 1)
stage_dir = stages[0]
self.assertTrue(stage_dir is not None)
cache_dir = stage_dir.outs[0].cache
info_dir = stage_dir.outs[0].info
info_dir = stage_dir.outs[0].dumpd()
md5_dir = info_dir['md5']
# Check status
sleep()
status = self.cloud.status([info])
self.assertEqual([(md5, STATUS_NEW)], status)
with self.cloud.project.state:
# Check status
sleep()
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_NEW)], status)
status_dir = self.cloud.status([info_dir])
self.assertTrue((md5_dir, STATUS_NEW) in status_dir)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertTrue((md5_dir, STATUS_NEW) in status_dir)
# Push and check status
self.cloud.push([info])
self.assertTrue(os.path.exists(cache))
self.assertTrue(os.path.isfile(cache))
# Push and check status
self.cloud.push([info])
self.assertTrue(os.path.exists(cache))
self.assertTrue(os.path.isfile(cache))
self.cloud.push([info_dir])
self.assertTrue(os.path.isfile(cache_dir))
self.cloud.push([info_dir])
self.assertTrue(os.path.isfile(cache_dir))
sleep()
status = self.cloud.status([info])
self.assertEqual([(md5, STATUS_OK)], status)
sleep()
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_OK)], status)
status_dir = self.cloud.status([info_dir])
self.assertTrue((md5_dir, STATUS_OK) in status_dir)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertTrue((md5_dir, STATUS_OK) in status_dir)
# Remove and check status
sleep()
shutil.rmtree(self.dvc.cache.local.cache_dir)
# Remove and check status
sleep()
shutil.rmtree(self.dvc.cache.local.cache_dir)
status = self.cloud.status([info])
self.assertEqual([(md5, STATUS_DELETED)], status)
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_DELETED)], status)
status_dir = self.cloud.status([info_dir])
self.assertEqual([(md5_dir, STATUS_DELETED)], status_dir)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertEqual([(md5_dir, STATUS_DELETED)], status_dir)
# Pull and check status
self.cloud.pull([info])
self.assertTrue(os.path.exists(cache))
self.assertTrue(os.path.isfile(cache))
with open(cache, 'r') as fd:
self.assertEqual(fd.read(), self.FOO_CONTENTS)
# Pull and check status
self.cloud.pull([info])
self.assertTrue(os.path.exists(cache))
self.assertTrue(os.path.isfile(cache))
with open(cache, 'r') as fd:
self.assertEqual(fd.read(), self.FOO_CONTENTS)
self.cloud.pull([info_dir])
self.assertTrue(os.path.isfile(cache_dir))
self.cloud.pull([info_dir])
self.assertTrue(os.path.isfile(cache_dir))
sleep()
status = self.cloud.status([info])
self.assertEqual([(md5, STATUS_OK)], status)
sleep()
status = self.cloud.status([info], show_checksums=True)
self.assertEqual([(md5, STATUS_OK)], status)
status_dir = self.cloud.status([info_dir])
self.assertTrue((md5_dir, STATUS_OK) in status_dir)
status_dir = self.cloud.status([info_dir], show_checksums=True)
self.assertTrue((md5_dir, STATUS_OK) in status_dir)
def test(self):
if self._should_test():
self._test_cloud()
self._ensure_should_run()
self._test_cloud()
class TestRemoteS3(TestDataCloudBase):
@@ -242,6 +306,17 @@ class TestRemoteGS(TestDataCloudBase):
return RemoteGS
class TestRemoteAzure(TestDataCloudBase):
def _should_test(self):
return _should_test_azure()
def _get_url(self):
return get_azure_url()
def _get_cloud_class(self):
return RemoteAzure
class TestRemoteLOCAL(TestDataCloudBase):
def _should_test(self):
return True
@@ -268,9 +343,6 @@ class TestRemoteSSH(TestDataCloudBase):
def _get_cloud_class(self):
return RemoteSSH
def test(self):
super(TestRemoteSSH, self).test()
class TestRemoteHDFS(TestDataCloudBase):
def _should_test(self):
@@ -282,9 +354,6 @@ class TestRemoteHDFS(TestDataCloudBase):
def _get_cloud_class(self):
return RemoteHDFS
def test(self):
super(TestRemoteHDFS, self).test()
class TestDataCloudCLIBase(TestDvc):
def main(self, args):
@@ -298,15 +367,21 @@ class TestDataCloudCLIBase(TestDvc):
else:
args += []
stage = self.dvc.add(self.FOO)
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertTrue(stage is not None)
cache = stage.outs[0].cache
stage_dir = self.dvc.add(self.DATA_DIR)
stages = self.dvc.add(self.DATA_DIR)
self.assertEqual(len(stages), 1)
stage_dir = stages[0]
self.assertTrue(stage_dir is not None)
cache_dir = stage_dir.outs[0].cache
#FIXME check status output
# FIXME check status output
sleep()
self.main(['status', '-c'] + args)
self.main(['status', '-c', '--show-checksums'] + args)
self.main(['push'] + args)
self.assertTrue(os.path.exists(cache))
@@ -314,23 +389,31 @@ class TestDataCloudCLIBase(TestDvc):
self.assertTrue(os.path.isfile(cache_dir))
sleep()
self.main(['status', '-c'] + args)
self.main(['status', '-c', '--show-checksums'] + args)
shutil.rmtree(self.dvc.cache.local.cache_dir)
sleep()
self.main(['status', '-c'] + args)
self.main(['status', '-c', '--show-checksums'] + args)
self.main(['fetch'] + args)
self.assertTrue(os.path.exists(cache))
self.assertTrue(os.path.isfile(cache))
self.assertTrue(os.path.isfile(cache_dir))
self.main(['pull'] + args)
self.assertTrue(os.path.exists(cache))
self.assertTrue(os.path.isfile(cache))
self.assertTrue(os.path.isfile(cache_dir))
self.assertTrue(os.path.isfile(self.FOO))
self.assertTrue(os.path.isdir(self.DATA_DIR))
sleep()
with open(cache, 'r') as fd:
self.assertEqual(fd.read(), self.FOO_CONTENTS)
self.assertTrue(os.path.isfile(cache_dir))
self.main(['status', '-c'] + args)
self.main(['status', '-c', '--show-checksums'] + args)
def _should_test(self):
return True
@@ -433,6 +516,18 @@ class TestRemoteGSCLI(TestDataCloudCLIBase):
self._test_cloud(TEST_REMOTE)
class TestRemoteAzureCLI(TestDataCloudCLIBase):
def _should_test(self):
return _should_test_azure()
def _test(self):
url = get_azure_url()
self.main(['remote', 'add', TEST_REMOTE, url])
self._test_cloud(TEST_REMOTE)
class TestDataCloudErrorCLI(TestDvc):
def main_fail(self, args):
ret = main(args)
+10
View File
@@ -0,0 +1,10 @@
from dvc.dependency import _get
from dvc.exceptions import DvcException
from tests.basic_env import TestDvc
class TestUnsupported(TestDvc):
def test(self):
with self.assertRaises(DvcException):
_get(None, 'unsupported://a/b', None)
+21
View File
@@ -0,0 +1,21 @@
import os
from dvc.main import main
from tests.test_repro import TestRepro
class TestDestroyNoConfirmation(TestRepro):
def test(self):
ret = main(['destroy'])
self.assertNotEqual(ret, 0)
class TestDestroyForce(TestRepro):
def test(self):
ret = main(['destroy', '-f'])
self.assertEqual(ret, 0)
self.assertFalse(os.path.exists(self.dvc.dvc_dir))
self.assertFalse(os.path.exists(self.file1_stage))
self.assertFalse(os.path.exists(self.file1))
+2 -1
View File
@@ -29,7 +29,8 @@ class TestGC(TestDvc):
self._test_gc()
def test_cli(self):
ret = main(['gc'])
ret = main(['gc', '-f'])
self.assertEqual(ret, 0)
self._test_gc()
def _test_gc(self):
+5 -1
View File
@@ -1,10 +1,10 @@
import os
import stat
import shutil
import filecmp
from dvc.main import main
from dvc.command.imp import CmdImport
from dvc.exceptions import DvcException
from tests.basic_env import TestDvc
@@ -18,3 +18,7 @@ class TestCmdImport(TestDvc):
ret = main(['import',
'non-existing-file', 'import'])
self.assertNotEqual(ret, 0)
def test_unsupported(self):
ret = main(['import', 'unsupported://path', 'import_unsupported'])
self.assertNotEqual(ret, 0)
+15
View File
@@ -24,6 +24,21 @@ class TestInit(TestGit):
self._test_init()
class TestDoubleInit(TestInit):
def test(self):
ret = main(['init'])
self.assertEqual(ret, 0)
self._test_init()
ret = main(['init'])
self.assertNotEqual(ret, 0)
self._test_init()
ret = main(['init', '--force'])
self.assertEqual(ret, 0)
self._test_init()
class TestInitNoSCMFail(TestDir):
def test_api(self):
with self.assertRaises(InitError):
+17
View File
@@ -1,6 +1,7 @@
import colorama
import logging
from dvc.command.base import CmdBase
from dvc.config import Config
from dvc.logger import Logger
@@ -32,3 +33,19 @@ class TestLogger(TestDvc):
msg = Logger.colorize(name, name)
# This is not a tty, so it should not colorize anything
self.assertEqual(msg, name)
class TestLoggerCLI(TestDvc):
def test(self):
class A(object):
quiet = True
verbose = False
args = A()
CmdBase._set_loglevel(args)
self.assertEqual(Logger.logger().getEffectiveLevel(), logging.CRITICAL)
args.quiet = False
args.verbose = True
CmdBase._set_loglevel(args)
self.assertEqual(Logger.logger().getEffectiveLevel(), logging.DEBUG)
+167 -10
View File
@@ -1,7 +1,9 @@
import os
import json
from dvc.project import Project
from dvc.main import main
from dvc.exceptions import DvcException
from tests.basic_env import TestDvc
@@ -19,6 +21,13 @@ class TestMetrics(TestDvc):
with open('metric_json', 'w+') as fd:
json.dump({'branch': branch}, fd)
with open('metric_csv', 'w+') as fd:
fd.write(branch)
with open('metric_hcsv', 'w+') as fd:
fd.write('branch\n')
fd.write(branch)
with open('metric_tsv', 'w+') as fd:
fd.write(branch)
@@ -26,7 +35,7 @@ class TestMetrics(TestDvc):
fd.write('branch\n')
fd.write(branch)
self.dvc.scm.add(['metric', 'metric_json', 'metric_tsv', 'metric_htsv'])
self.dvc.scm.add(['metric', 'metric_json', 'metric_tsv', 'metric_htsv', 'metric_csv', 'metric_hcsv'])
self.dvc.scm.commit('metric')
self.dvc.scm.checkout('master')
@@ -38,34 +47,57 @@ class TestMetrics(TestDvc):
self.assertTrue(ret['bar']['metric'] == 'bar')
self.assertTrue(ret['baz']['metric'] == 'baz')
ret = self.dvc.metrics_show('metric_json', json_path='branch', all_branches=True)
ret = self.dvc.metrics_show('metric_json',
typ='json',
xpath='branch',
all_branches=True)
self.assertEqual(len(ret), 3)
self.assertTrue(ret['foo']['metric_json'] == ['foo'])
self.assertTrue(ret['bar']['metric_json'] == ['bar'])
self.assertTrue(ret['baz']['metric_json'] == ['baz'])
ret = self.dvc.metrics_show('metric_tsv', tsv_path='0,0', all_branches=True)
ret = self.dvc.metrics_show('metric_tsv',
typ='tsv',
xpath='0,0',
all_branches=True)
self.assertEqual(len(ret), 3)
self.assertTrue(ret['foo']['metric_tsv'] == ['foo'])
self.assertTrue(ret['bar']['metric_tsv'] == ['bar'])
self.assertTrue(ret['baz']['metric_tsv'] == ['baz'])
ret = self.dvc.metrics_show('metric_htsv', htsv_path='branch,0', all_branches=True)
ret = self.dvc.metrics_show('metric_htsv',
typ='htsv',
xpath='branch,0',
all_branches=True)
self.assertEqual(len(ret), 3)
self.assertTrue(ret['foo']['metric_htsv'] == ['foo'])
self.assertTrue(ret['bar']['metric_htsv'] == ['bar'])
self.assertTrue(ret['baz']['metric_htsv'] == ['baz'])
ret = self.dvc.metrics_show('metric_csv',
typ='csv',
xpath='0,0',
all_branches=True)
self.assertEqual(len(ret), 3)
self.assertTrue(ret['foo']['metric_csv'] == ['foo'])
self.assertTrue(ret['bar']['metric_csv'] == ['bar'])
self.assertTrue(ret['baz']['metric_csv'] == ['baz'])
ret = self.dvc.metrics_show('metric_hcsv',
typ='hcsv',
xpath='branch,0',
all_branches=True)
self.assertEqual(len(ret), 3)
self.assertTrue(ret['foo']['metric_hcsv'] == ['foo'])
self.assertTrue(ret['bar']['metric_hcsv'] == ['bar'])
self.assertTrue(ret['baz']['metric_hcsv'] == ['baz'])
class TestMetricsReproCLI(TestDvc):
def test(self):
stage = self.dvc.run(outs_no_cache=['metrics'],
stage = self.dvc.run(metrics_no_cache=['metrics'],
cmd='python {} {} {}'.format(self.CODE, self.FOO, 'metrics'))
ret = main(['metrics', 'add', 'metrics'])
self.assertEqual(ret, 0)
ret = main(['repro', '-m', stage.path])
self.assertEqual(ret, 0)
@@ -73,10 +105,32 @@ class TestMetricsReproCLI(TestDvc):
self.assertEqual(ret, 0)
ret = main(['repro', '-f', '-m', stage.path])
self.assertNotEqual(ret, 0)
ret = main(['metrics', 'add', 'metrics'])
self.assertEqual(ret, 0)
ret = main(['metrics', 'modify', '-t', 'csv', '-x', '0,0', 'metrics'])
self.assertEqual(ret, 0)
class TestMetricsCLI(TestMetrics):
ret = main(['repro', '-f', '-m', stage.path])
self.assertEqual(ret, 0)
def test_dir(self):
os.mkdir('metrics_dir')
with self.assertRaises(DvcException):
self.dvc.run(metrics_no_cache=['metrics_dir'])
def test_binary(self):
with open('metrics_bin', 'w+') as fd:
fd.write('\0\0\0\0\0\0\0\0')
with self.assertRaises(DvcException):
self.dvc.run(metrics_no_cache=['metrics_bin'])
class TestMetricsCLICompat(TestMetrics):
def test(self):
#FIXME check output
ret = main(['metrics', 'show', '--all-branches', 'metric', '-v'])
@@ -90,3 +144,106 @@ class TestMetricsCLI(TestMetrics):
ret = main(['metrics', 'show', '--all-branches', 'metric_htsv', '--htsv-path', 'branch,0'])
self.assertEqual(ret, 0)
ret = main(['metrics', 'show', '--all-branches', 'metric_csv', '--csv-path', '0,0'])
self.assertEqual(ret, 0)
ret = main(['metrics', 'show', '--all-branches', 'metric_hcsv', '--hcsv-path', 'branch,0'])
self.assertEqual(ret, 0)
class TestMetricsCLI(TestMetrics):
def test(self):
#FIXME check output
ret = main(['metrics', 'show', '-a', 'metric', '-v'])
self.assertEqual(ret, 0)
ret = main(['metrics', 'show', '-a', 'metric_json', '-t', 'json',
'-x', 'branch'])
self.assertEqual(ret, 0)
ret = main(['metrics', 'show', '-a', 'metric_tsv', '-t', 'tsv',
'-x', '0,0'])
self.assertEqual(ret, 0)
ret = main(['metrics', 'show', '-a', 'metric_htsv', '-t', 'htsv',
'-x', 'branch,0'])
self.assertEqual(ret, 0)
ret = main(['metrics', 'show', '-a', 'metric_csv', '-t', 'csv',
'-x', '0,0'])
self.assertEqual(ret, 0)
ret = main(['metrics', 'show', '-a', 'metric_hcsv', '-t', 'hcsv',
'-x', 'branch,0'])
self.assertEqual(ret, 0)
def test_dir(self):
os.mkdir('metrics_dir')
with self.assertRaises(DvcException):
self.dvc.run(outs_no_cache=['metrics_dir'])
self.dvc.metrics_add('metrics_dir')
def test_binary(self):
with open('metrics_bin', 'w+') as fd:
fd.write('\0\0\0\0\0\0\0\0')
with self.assertRaises(DvcException):
self.dvc.run(outs_no_cache=['metrics_bin'])
self.dvc.metrics_add('metrics_bin')
def test_non_existing(self):
ret = main(['metrics', 'add', 'non-existing'])
self.assertNotEqual(ret, 0)
ret = main(['metrics', 'modify', 'non-existing'])
self.assertNotEqual(ret, 0)
ret = main(['metrics', 'remove', 'non-existing'])
self.assertNotEqual(ret, 0)
class TestNoMetrics(TestDvc):
def test(self):
ret = main(['metrics', 'show'])
self.assertNotEqual(ret, 0)
class TestCachedMetrics(TestDvc):
def _do_write(self, branch):
self.dvc.scm.checkout(branch)
self.dvc.checkout()
with open('metrics.json', 'w+') as fd:
json.dump({'metrics': branch}, fd)
stages = self.dvc.add('metrics.json')
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertNotEqual(stage, None)
self.dvc.scm.add(['.gitignore', 'metrics.json.dvc'])
self.dvc.scm.commit(branch)
def test(self):
self.dvc.scm.commit('init')
self.dvc.scm.branch('one')
self.dvc.scm.branch('two')
self._do_write('master')
self._do_write('one')
self._do_write('two')
self.dvc = Project('.')
res = self.dvc.metrics_show('metrics.json',
all_branches=True,
typ='json',
xpath='metrics')
self.assertEqual(res, {"master": {"metrics.json": ["master"]},
"one": {"metrics.json": ["one"]},
"two": {"metrics.json": ["two"]}})
+14 -1
View File
@@ -1,9 +1,10 @@
import os
from dvc.main import main
from dvc.exceptions import DvcException
from dvc.exceptions import DvcException, MoveNotDataSourceError
from tests.basic_env import TestDvc
from tests.test_repro import TestRepro
class TestMove(TestDvc):
@@ -39,3 +40,15 @@ class TestCmdMove(TestDvc):
ret = main(['move', 'non-existing-file', 'dst'])
self.assertNotEqual(ret, 0)
class TestMoveNotDataSource(TestRepro):
def test(self):
from dvc.project import Project
self.dvc = Project(self._root_dir)
with self.assertRaises(MoveNotDataSourceError):
self.dvc.move(self.file1, 'dst')
ret = main(['move', self.file1, 'dst'])
self.assertNotEqual(ret, 0)
+10
View File
@@ -0,0 +1,10 @@
from dvc.output import _get
from dvc.exceptions import DvcException
from tests.basic_env import TestDvc
class TestUnsupported(TestDvc):
def test(self):
with self.assertRaises(DvcException):
_get(None, 'unsupported://a/b', None, None, None)
+167
View File
@@ -0,0 +1,167 @@
from dvc.main import main
from tests.basic_env import TestDvc
from tests.test_repro import TestRepro, TestReproChangedDeepData
import os
class TestPipelineShowSingle(TestDvc):
def setUp(self):
super(TestPipelineShowSingle, self).setUp()
self.stage = 'foo.dvc'
self.dotFile = 'graph.dot'
ret = main(['add', self.FOO])
self.assertEqual(ret, 0)
def test(self):
ret = main(['pipeline', 'show', self.stage])
self.assertEqual(ret, 0)
def test_commands(self):
ret = main(['pipeline', 'show', self.stage, '--commands'])
self.assertEqual(ret, 0)
def test_outs(self):
ret = main(['pipeline', 'show', self.stage, '--outs'])
self.assertEqual(ret, 0)
def test_ascii(self):
ret = main(['pipeline', 'show', '--ascii', self.stage])
self.assertEqual(ret, 0)
def test_dot(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.stage])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_ascii_commands(self):
ret = main(['pipeline', 'show', '--ascii', self.stage, '--commands'])
self.assertEqual(ret, 0)
def test_ascii_outs(self):
ret = main(['pipeline', 'show', '--ascii', self.stage, '--outs'])
self.assertEqual(ret, 0)
def test_dot_commands(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.stage, '--commands'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_dot_outs(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.stage, '--outs'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_not_dvc_file(self):
ret = main(['pipeline', 'show', self.FOO])
self.assertNotEqual(ret, 0)
def test_non_existing(self):
ret = main(['pipeline', 'show', 'non-existing'])
self.assertNotEqual(ret, 0)
class TestPipelineShow(TestRepro):
def setUp(self):
super(TestPipelineShow, self).setUp()
self.dotFile = 'graph.dot'
def test(self):
ret = main(['pipeline', 'show', self.file1_stage])
self.assertEqual(ret, 0)
def test_commands(self):
ret = main(['pipeline', 'show', self.file1_stage, '--commands'])
self.assertEqual(ret, 0)
def test_outs(self):
ret = main(['pipeline', 'show', self.file1_stage, '--outs'])
self.assertEqual(ret, 0)
def test_ascii(self):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage])
self.assertEqual(ret, 0)
def test_dot(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_ascii_commands(self):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage, '--commands'])
self.assertEqual(ret, 0)
def test_ascii_outs(self):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage, '--outs'])
self.assertEqual(ret, 0)
def test_dot_commands(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage, '--commands'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_dot_outs(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage, '--outs'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_not_dvc_file(self):
ret = main(['pipeline', 'show', self.file1])
self.assertNotEqual(ret, 0)
def test_non_existing(self):
ret = main(['pipeline', 'show', 'non-existing'])
self.assertNotEqual(ret, 0)
class TestPipelineShowDeep(TestReproChangedDeepData):
def setUp(self):
super(TestPipelineShowDeep, self).setUp()
self.dotFile = 'graph.dot'
def test(self):
ret = main(['pipeline', 'show', self.file1_stage])
self.assertEqual(ret, 0)
def test_commands(self):
ret = main(['pipeline', 'show', self.file1_stage, '--commands'])
self.assertEqual(ret, 0)
def test_outs(self):
ret = main(['pipeline', 'show', self.file1_stage, '--outs'])
self.assertEqual(ret, 0)
def test_ascii(self):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage])
self.assertEqual(ret, 0)
def test_dot(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_ascii_commands(self):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage, '--commands'])
self.assertEqual(ret, 0)
def test_ascii_outs(self):
ret = main(['pipeline', 'show', '--ascii', self.file1_stage, '--outs'])
self.assertEqual(ret, 0)
def test_dot_commands(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage, '--commands'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_dot_outs(self):
ret = main(['pipeline', 'show', '--dot', self.dotFile, self.file1_stage, '--outs'])
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(self.dotFile))
def test_not_dvc_file(self):
ret = main(['pipeline', 'show', self.file1])
self.assertNotEqual(ret, 0)
def test_non_existing(self):
ret = main(['pipeline', 'show', 'non-existing'])
self.assertNotEqual(ret, 0)
+66
View File
@@ -1,4 +1,12 @@
import os
import configobj
from dvc.main import main
from dvc.project import Project
from dvc.command.remote import CmdRemoteAdd
from dvc.exceptions import UnsupportedRemoteError
from dvc.remote import Remote
from dvc.config import Config
from tests.basic_env import TestDvc
@@ -19,3 +27,61 @@ class TestRemote(TestDvc):
self.assertEqual(main(['remote', 'modify', remotes[0], 'option', 'value']), 0)
self.assertEqual(main(['remote', 'list']), 0)
def test_failed_write(self):
class A(object):
local = False
name = 'myremote'
url = 's3://remote'
unset = False
args = A()
cmd = CmdRemoteAdd(args)
cmd.configobj.write = None
ret = cmd.run()
self.assertNotEqual(ret, 0)
def test_unsupported(self):
with self.assertRaises(UnsupportedRemoteError):
Remote(self.dvc, {Config.SECTION_REMOTE_URL: 'unsupported://url'})
class TestRemoteRemoveDefault(TestDvc):
def test(self):
remote = 'mys3'
self.assertEqual(main(['remote', 'add', '--default', remote, 's3://bucket/name']), 0)
self.assertEqual(main(['remote', 'modify', remote, 'profile', 'default']), 0)
self.assertEqual(main(['config', '--local', 'core.remote', remote]), 0)
config = configobj.ConfigObj(os.path.join(self.dvc.dvc_dir, Config.CONFIG))
local_config = configobj.ConfigObj(os.path.join(self.dvc.dvc_dir, Config.CONFIG_LOCAL))
self.assertEqual(config[Config.SECTION_CORE][Config.SECTION_CORE_REMOTE], remote)
self.assertEqual(local_config[Config.SECTION_CORE][Config.SECTION_CORE_REMOTE], remote)
self.assertEqual(main(['remote', 'remove', remote]), 0)
config = configobj.ConfigObj(os.path.join(self.dvc.dvc_dir, Config.CONFIG))
local_config = configobj.ConfigObj(os.path.join(self.dvc.dvc_dir, Config.CONFIG_LOCAL))
section = Config.SECTION_REMOTE_FMT.format(remote)
self.assertTrue(section not in config.keys())
core = config.get(Config.SECTION_CORE, None)
if core is not None:
self.assertTrue(Config.SECTION_CORE_REMOTE not in core.keys())
core = local_config.get(Config.SECTION_CORE, None)
if core is not None:
self.assertTrue(Config.SECTION_CORE_REMOTE not in core.keys())
class TestRemoteRemove(TestDvc):
def test(self):
ret = main(['config', 'core.jobs', '1'])
self.assertEqual(ret, 0)
remote = 'mys3'
ret = main(['remote', 'add', remote, 's3://bucket/name'])
self.assertEqual(ret, 0)
ret = main(['remote', 'remove', remote])
self.assertEqual(ret, 0)
+14 -6
View File
@@ -1,8 +1,7 @@
import os
from dvc.main import main
from dvc.stage import Stage
from dvc.project import NotDvcFileError
from dvc.stage import Stage, StageFileDoesNotExistError
from dvc.command.remove import CmdRemove
from tests.basic_env import TestDvc
@@ -10,7 +9,10 @@ from tests.basic_env import TestDvc
class TestRemove(TestDvc):
def test(self):
stage = self.dvc.add(self.FOO)
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertTrue(stage is not None)
stage_removed = self.dvc.remove(stage.path, outs_only=True)
self.assertIsInstance(stage_removed, Stage)
@@ -26,13 +28,16 @@ class TestRemove(TestDvc):
class TestRemoveNonExistentFile(TestDvc):
def test(self):
with self.assertRaises(NotDvcFileError) as cx:
with self.assertRaises(StageFileDoesNotExistError) as cx:
self.dvc.remove('non_existent_dvc_file')
class TestRemoveDirectory(TestDvc):
def test(self):
stage_add = self.dvc.add(self.DATA_DIR)
stages = self.dvc.add(self.DATA_DIR)
self.assertEqual(len(stages), 1)
stage_add = stages[0]
self.assertTrue(stage_add is not None)
stage_removed = self.dvc.remove(stage_add.path)
self.assertEqual(stage_add.path, stage_removed.path)
self.assertFalse(os.path.exists(self.DATA_DIR))
@@ -41,7 +46,10 @@ class TestRemoveDirectory(TestDvc):
class TestCmdRemove(TestDvc):
def test(self):
stage = self.dvc.add(self.FOO)
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertTrue(stage is not None)
ret = main(['remove',
stage.path])
self.assertEqual(ret, 0)
+218 -29
View File
@@ -14,10 +14,11 @@ from google.cloud import storage as gc
from dvc.main import main
from dvc.command.repro import CmdRepro
from dvc.project import Project, ReproductionError, NotDvcFileError
from dvc.project import Project, ReproductionError
from dvc.utils import file_md5
from dvc.remote.local import RemoteLOCAL
from dvc.stage import Stage
from dvc.stage import Stage, StageFileDoesNotExistError
from dvc.system import System
from tests.basic_env import TestDvc
from tests.test_data_cloud import _should_test_aws, TEST_AWS_REPO_BUCKET
@@ -30,7 +31,10 @@ class TestRepro(TestDvc):
def setUp(self):
super(TestRepro, self).setUp()
self.foo_stage = self.dvc.add(self.FOO)
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
self.foo_stage = stages[0]
self.assertTrue(self.foo_stage is not None)
self.file1 = 'file1'
self.file1_stage = self.file1 + '.dvc'
@@ -40,9 +44,20 @@ class TestRepro(TestDvc):
cmd='python {} {} {}'.format(self.CODE, self.FOO, self.file1))
class TestReproFail(TestRepro):
def test(self):
os.unlink(self.CODE)
ret = main(['repro', self.file1_stage])
self.assertNotEqual(ret, 0)
class TestReproDepUnderDir(TestDvc):
def test(self):
self.dir_stage = self.dvc.add(self.DATA_DIR)
stages = self.dvc.add(self.DATA_DIR)
self.assertEqual(len(stages), 1)
self.dir_stage = stages[0]
self.assertTrue(self.dir_stage is not None)
sleep()
@@ -53,7 +68,7 @@ class TestReproDepUnderDir(TestDvc):
deps=[self.DATA, self.CODE],
cmd='python {} {} {}'.format(self.CODE, self.DATA, self.file1))
self.assertTrue(filecmp.cmp(self.file1, self.DATA))
self.assertTrue(filecmp.cmp(self.file1, self.DATA, shallow=False))
os.unlink(self.DATA)
shutil.copyfile(self.FOO, self.DATA)
@@ -62,7 +77,7 @@ class TestReproDepUnderDir(TestDvc):
stages = self.dvc.reproduce(self.file1_stage)
self.assertEqual(len(stages), 2)
self.assertTrue(filecmp.cmp(self.file1, self.FOO))
self.assertTrue(filecmp.cmp(self.file1, self.FOO, shallow=False))
class TestReproNoDeps(TestRepro):
@@ -93,7 +108,7 @@ class TestReproChangedCode(TestRepro):
stages = self.dvc.reproduce(self.file1_stage)
self.assertTrue(filecmp.cmp(self.file1, self.BAR))
self.assertTrue(filecmp.cmp(self.file1, self.BAR, shallow=False))
self.assertEqual(len(stages), 1)
def swap_code(self):
@@ -109,7 +124,7 @@ class TestReproChangedData(TestRepro):
stages = self.dvc.reproduce(self.file1_stage)
self.assertTrue(filecmp.cmp(self.file1, self.BAR))
self.assertTrue(filecmp.cmp(self.file1, self.BAR, shallow=False))
self.assertEqual(len(stages), 2)
def swap_foo_with_bar(self):
@@ -117,24 +132,140 @@ class TestReproChangedData(TestRepro):
shutil.copyfile(self.BAR, self.FOO)
class TestReproChangedDeepData(TestReproChangedData):
class TestReproDry(TestReproChangedData):
def test(self):
file2 = 'file2'
file2_stage = file2 + '.dvc'
self.dvc.run(fname=file2_stage,
outs=[file2],
deps=[self.file1, self.CODE],
cmd='python {} {} {}'.format(self.CODE, self.file1, file2))
self.swap_foo_with_bar()
stages = self.dvc.reproduce(file2_stage)
stages = self.dvc.reproduce(self.file1_stage, dry=True)
self.assertTrue(filecmp.cmp(self.file1, self.BAR))
self.assertTrue(filecmp.cmp(file2, self.BAR))
self.assertTrue(len(stages), 2)
self.assertFalse(filecmp.cmp(self.file1, self.BAR, shallow=False))
ret = main(['repro', '--dry', self.file1_stage])
self.assertEqual(ret, 0)
self.assertFalse(filecmp.cmp(self.file1, self.BAR, shallow=False))
class TestReproUpToDate(TestRepro):
def test(self):
ret = main(['repro', self.file1_stage])
self.assertEqual(ret, 0)
class TestReproDryNoExec(TestDvc):
def test(self):
deps = []
for d in range(3):
idir = 'idir{}'.format(d)
odir = 'odir{}'.format(d)
deps.append('-d')
deps.append(odir)
os.mkdir(idir)
f = os.path.join(idir, 'file')
with open(f, 'w+') as fobj:
fobj.write(str(d))
if os.name == 'nt':
cp = 'copy'
else:
cp = 'cp'
ret = main(['run',
'--no-exec',
'-d', idir,
'-o', odir,
"python -c 'import shutil; "
"shutil.copytree(\"{}\", \"{}\")'".format(idir, odir)])
self.assertEqual(ret, 0)
ret = main(['run', '--no-exec', '-f', 'Dvcfile'] + deps)
self.assertEqual(ret, 0)
ret = main(['repro', '--dry'])
self.assertEqual(ret, 0)
class TestReproChangedDeepData(TestReproChangedData):
def setUp(self):
super(TestReproChangedDeepData, self).setUp()
self.file2 = 'file2'
self.file2_stage = self.file2 + '.dvc'
self.dvc.run(fname=self.file2_stage,
outs=[self.file2],
deps=[self.file1, self.CODE],
cmd='python {} {} {}'.format(self.CODE,
self.file1,
self.file2))
def test(self):
self.swap_foo_with_bar()
stages = self.dvc.reproduce(self.file2_stage)
self.assertTrue(filecmp.cmp(self.file1, self.BAR, shallow=False))
self.assertTrue(filecmp.cmp(self.file2, self.BAR, shallow=False))
self.assertEqual(len(stages), 3)
class TestReproPipeline(TestReproChangedDeepData):
def test(self):
stages = self.dvc.reproduce(self.file1_stage,
force=True,
pipeline=True)
self.assertEqual(len(stages), 3)
def test_cli(self):
ret = main(['repro', '--pipeline', '-f', self.file1_stage])
self.assertEqual(ret, 0)
class TestReproPipelines(TestDvc):
def setUp(self):
super(TestReproPipelines, self).setUp()
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
self.foo_stage = stages[0]
self.assertTrue(self.foo_stage is not None)
stages = self.dvc.add(self.BAR)
self.assertEqual(len(stages), 1)
self.bar_stage = stages[0]
self.assertTrue(self.bar_stage is not None)
self.file1 = 'file1'
self.file1_stage = self.file1 + '.dvc'
self.dvc.run(fname=self.file1_stage,
outs=[self.file1],
deps=[self.FOO, self.CODE],
cmd='python {} {} {}'.format(self.CODE, self.FOO, self.file1))
self.file2 = 'file2'
self.file2_stage = self.file2 + '.dvc'
self.dvc.run(fname=self.file2_stage,
outs=[self.file2],
deps=[self.BAR, self.CODE],
cmd='python {} {} {}'.format(self.CODE, self.BAR, self.file2))
def test(self):
stages = self.dvc.reproduce(all_pipelines=True, force=True)
self.assertEqual(len(stages), 4)
names = [stage.relpath for stage in stages]
self.assertTrue(self.foo_stage.relpath in names)
self.assertTrue(self.bar_stage.relpath in names)
self.assertTrue(self.file1_stage in names)
self.assertTrue(self.file2_stage in names)
def test_cli(self):
ret = main(['repro', '-f', '-P'])
self.assertEqual(ret, 0)
class TestReproLocked(TestReproChangedData):
def test(self):
file2 = 'file2'
@@ -154,12 +285,12 @@ class TestReproLocked(TestReproChangedData):
ret = main(['unlock', file2_stage])
self.assertEqual(ret, 0)
stages = self.dvc.reproduce(file2_stage)
self.assertTrue(filecmp.cmp(self.file1, self.BAR))
self.assertTrue(filecmp.cmp(file2, self.BAR))
self.assertTrue(filecmp.cmp(self.file1, self.BAR, shallow=False))
self.assertTrue(filecmp.cmp(file2, self.BAR, shallow=False))
self.assertEqual(len(stages), 3)
def test_non_existing(self):
with self.assertRaises(NotDvcFileError):
with self.assertRaises(StageFileDoesNotExistError):
self.dvc.lock_stage('non-existing-stage')
ret = main(['lock', 'non-existing-stage'])
@@ -174,7 +305,7 @@ class TestReproPhony(TestReproChangedData):
self.dvc.reproduce(stage.path)
self.assertTrue(filecmp.cmp(self.file1, self.BAR))
self.assertTrue(filecmp.cmp(self.file1, self.BAR, shallow=False))
class TestNonExistingOutput(TestRepro):
@@ -191,7 +322,7 @@ class TestReproDataSource(TestReproChangedData):
stages = self.dvc.reproduce(self.foo_stage.path)
self.assertTrue(filecmp.cmp(self.FOO, self.BAR))
self.assertTrue(filecmp.cmp(self.FOO, self.BAR, shallow=False))
self.assertEqual(stages[0].outs[0].md5, file_md5(self.BAR)[0])
@@ -222,6 +353,55 @@ class TestReproChangedDir(TestDvc):
self.assertEqual(len(stages), 1)
class TestReproChangedDirData(TestDvc):
def test(self):
dir_name = 'dir'
dir_code = 'dir_code.py'
sleep()
with open(dir_code, 'w+') as fd:
fd.write("import os; import sys; import shutil; shutil.copytree(sys.argv[1], sys.argv[2])")
sleep()
stage = self.dvc.run(outs=[dir_name],
deps=[self.DATA_DIR, dir_code],
cmd="python {} {} {}".format(dir_code,
self.DATA_DIR,
dir_name))
self.assertTrue(stage is not None)
stages = self.dvc.reproduce(stage.path)
self.assertEqual(len(stages), 0)
sleep()
with open(self.DATA_SUB, 'a') as fd:
fd.write('add')
sleep()
stages = self.dvc.reproduce(stage.path)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
sleep()
# Check that dvc indeed registers changed output dir
shutil.move(self.BAR, dir_name)
sleep()
stages = self.dvc.reproduce(stage.path)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
# Check that dvc registers mtime change for the directory.
System.hardlink(self.DATA_SUB, self.DATA_SUB + '.lnk')
sleep()
stages = self.dvc.reproduce(stage.path)
self.assertEqual(len(stages), 1)
self.assertTrue(stages[0] is not None)
class TestReproMissingMd5InStageFile(TestRepro):
def test(self):
with open(self.file1_stage, 'r') as fd:
@@ -237,8 +417,13 @@ class TestReproMissingMd5InStageFile(TestRepro):
self.assertEqual(len(stages), 1)
class TestCmdRepro(TestRepro):
class TestCmdRepro(TestReproChangedData):
def test(self):
self.swap_foo_with_bar()
ret = main(['status'])
self.assertEqual(ret, 0)
ret = main(['repro',
self.file1_stage])
self.assertEqual(ret, 0)
@@ -267,7 +452,7 @@ class TestCmdReproChdir(TestDvc):
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(foo))
self.assertTrue(os.path.isfile(bar))
self.assertTrue(filecmp.cmp(foo, bar))
self.assertTrue(filecmp.cmp(foo, bar, shallow=False))
os.unlink(bar)
@@ -276,7 +461,7 @@ class TestCmdReproChdir(TestDvc):
self.assertEqual(ret, 0)
self.assertTrue(os.path.isfile(foo))
self.assertTrue(os.path.isfile(bar))
self.assertTrue(filecmp.cmp(foo, bar))
self.assertTrue(filecmp.cmp(foo, bar, shallow=False))
class TestReproExternalBase(TestDvc):
@@ -335,7 +520,9 @@ class TestReproExternalBase(TestDvc):
import_stage = self.dvc.imp(out_foo_path, 'import')
self.assertTrue(os.path.exists('import'))
self.assertTrue(filecmp.cmp('import', self.FOO))
self.assertTrue(filecmp.cmp('import', self.FOO, shallow=False))
import_remote_stage = self.dvc.imp(out_foo_path, out_foo_path + '_imported')
cmd_stage = self.dvc.run(outs=[out_bar_path],
deps=[out_foo_path],
@@ -345,10 +532,12 @@ class TestReproExternalBase(TestDvc):
sleep()
self.dvc.status()
stages = self.dvc.reproduce(import_stage.path)
self.assertEqual(len(stages), 1)
self.assertTrue(os.path.exists('import'))
self.assertTrue(filecmp.cmp('import', self.BAR))
self.assertTrue(filecmp.cmp('import', self.BAR, shallow=False))
stages = self.dvc.reproduce(cmd_stage.path)
self.assertEqual(len(stages), 1)

Some files were not shown because too many files have changed in this diff Show More