Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-10 07:18:27 +01:00
Code Issues Packages Projects Releases Wiki Activity
986 Commits 7 Branches 579 Tags
5fa796d95cdae7f1e04feb42b090cb0a10d8d774
Commit Graph
986 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev 5fa796d95c Merge pull request #537 from efiop/master 
checkout: optimize lost hardlink search
 2018-03-10 13:14:26 +02:00
Ruslan Kuprieiev dce1935651 checkout: optimize lost hardlink search 
Speeds up checkout for 30K files from 6sec to 3.5sec

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-10 13:10:15 +02:00
Ruslan Kuprieiev 066151617e Merge pull request #536 from efiop/master 
dvc: calculate 'effective inode' from getfileinfo() on windows
 2018-03-10 12:47:38 +02:00
Ruslan Kuprieiev 7094ec0151 dvc: calculate 'effective inode' from getfileinfo() on windows 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-10 12:40:46 +02:00
Ruslan Kuprieiev 4cd789bf2a Merge pull request #532 from efiop/master 
dvc: switch from yaml to json for dir cache file
 2018-03-09 08:40:07 +02:00
Ruslan Kuprieiev 3b9259ff17 dvc: switch from yaml to json for dir cache file 
yaml is extremely slow without CLoader, which caused our checkout()
and similar functions to take way too long to process dir cache files.
These files are not really meant to be read by the users, so we can
sacrifice the readability if yaml for better performance of json module.

E.g. checkout for 30K files used to take 30sec and after this patch only
takes 6sec, which is still quite slow, but much more bearable.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-09 08:22:44 +02:00
Ruslan Kuprieiev a32ac41e3c Merge pull request #531 from efiop/master 
Fixes for pull/push CLI and cache
 2018-03-08 22:48:32 +02:00
Ruslan Kuprieiev 54a6532360 cache: create cache dirs only when needed 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-08 19:45:58 +02:00
Ruslan Kuprieiev 2e21c39c00 cli: move push/pull from under 'dvc cache' 
Fixes #528

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-08 14:03:49 +02:00
Ruslan Kuprieiev 1248b56f44 Merge pull request #529 from efiop/master 
config and cloud fixes
 2018-03-08 13:27:22 +02:00
Ruslan Kuprieiev 4bc62cbf66 aws: don't forget to use relpath in progress callback 
Fixes #519

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-08 12:44:44 +02:00
Ruslan Kuprieiev ff77d9dc46 dvc: calculate n jobs as 8*n_cpu 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-08 09:51:24 +02:00
Ruslan Kuprieiev fa43309222 config: rename 'Global' section to 'Core' 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-08 09:04:28 +02:00
Ruslan Kuprieiev 2e83045d72 Merge pull request #526 from efiop/master 
repro: don't fail if dir cache is not present
 2018-03-08 08:44:17 +02:00
Ruslan Kuprieiev 4a1e20c219 repro: don't fail if dir cache is not present 
Fixes #518

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-08 08:38:13 +02:00
Ruslan Kuprieiev 8dff57eb9c Merge pull request #525 from efiop/master 
introduce 'dvc fetch' and change 'dvc remove' behavior
 2018-03-08 08:21:18 +02:00
Ruslan Kuprieiev 36be0a221a dvc: introduce 'dvc cache fetch' 
Fixes #521

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-08 07:08:45 +02:00
Ruslan Kuprieiev 783fad935b remove: use stage files as targets and only remove outputs 
Fixes #516

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-08 06:59:01 +02:00
Dmitry Petrov e359d1e538 Merge pull request #520 from dataversioncontrol/small_scripts_fixes 
Small scripts fixes
 2018-03-06 16:00:04 -08:00
Dmitry f122c19981 Check dir changes properly. Close #491. 2018-03-06 15:52:39 -08:00
Dmitry ec741d87d5 Remove EC2 params from config 2018-03-06 15:10:28 -08:00
Dmitry 30f031e5e2 CNN tutorial: use external code files 2018-03-06 15:09:51 -08:00
Dmitry Petrov dbace13ad4 Merge pull request #517 from dataversioncontrol/cnn_classify_tutorial_rename 
CNN tutorial: rename
 2018-03-05 00:39:07 -08:00
Dmitry 6fb7bd7825 CNN tutorial: rename 2018-03-05 00:11:03 -08:00
Ruslan Kuprieiev 6ab9b65884 Merge pull request #514 from efiop/master 
status: print paths relative to cache dir
 2018-03-04 16:09:08 +02:00
Ruslan Kuprieiev cece5c308c status: print paths relative to cache dir 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-04 16:04:32 +02:00
Ruslan Kuprieiev 6ce6615914 Merge pull request #513 from efiop/master 
checkout: don't forget to set perms back
 2018-03-04 15:57:00 +02:00
Ruslan Kuprieiev c1d35df630 checkout: don't forget to set perms back 
Fixes #505

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-04 15:50:17 +02:00
Ruslan Kuprieiev 974bdd8f38 Merge pull request #512 from efiop/master 
Pull/push per dvc-file
 2018-03-04 15:16:16 +02:00
Ruslan Kuprieiev bdd51c854b cloud: don't print progress bar when collecting dir 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-04 15:11:21 +02:00
Ruslan Kuprieiev dbc6e1ac88 dvc: add push/pull/status per dvc-file 
Fixes #453

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-04 15:10:33 +02:00
Ruslan Kuprieiev 0e25984ccc Merge pull request #511 from efiop/master 
checkout: use is_dir_cache
 2018-03-04 13:10:52 +02:00
Ruslan Kuprieiev 99f930b779 checkout: use is_dir_cache 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-04 12:58:59 +02:00
Dmitry Petrov a3030fd100 Merge pull request #510 from dataversioncontrol/cnn_classify_tutorial 
CNN tutorial
 2018-03-04 02:22:16 -08:00
Dmitry 4d994d1220 CNN tutorial 2018-03-04 01:48:21 -08:00
Ruslan Kuprieiev c95ffa29f3 Merge pull request #509 from efiop/master 
Single-file cache for dir structure
 2018-03-04 11:31:31 +02:00
Ruslan Kuprieiev 6f9c8eed91 cloud: use paths relative to cache_dir in progress bar 
Fixes #508

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-04 11:25:48 +02:00
Ruslan Kuprieiev 7f4c7783e7 dvc: store cached dir structure in a single json file 
Fixes #507

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-04 11:15:20 +02:00
Ruslan Kuprieiev ed11055c19 progress: reserve more space in a progress bar 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-04 11:02:40 +02:00
Dmitry Petrov 5bc7ff9867 Merge pull request #506 from dataversioncontrol/misc_fixes 
Misc fixes: Not expected files in dir cache and message formatting
 2018-03-03 12:20:03 -08:00
Dmitry e3b63be0c2 Messages formatting. Close #492 2018-03-03 12:11:03 -08:00
Dmitry 241dfed3ee Not expected files in dir cache 2018-03-03 11:42:45 -08:00
Ruslan Kuprieiev d0263aa500 Merge pull request #504 from efiop/master 
dvc: properly collect used cache
 2018-03-03 04:34:47 +02:00
Ruslan Kuprieiev f814269b4c dvc: properly collect used cache 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-03 04:25:46 +02:00
Ruslan Kuprieiev d80b7da3ef cloud: makedirs: catch exception instead of checking in advance 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-03 04:24:12 +02:00
Ruslan Kuprieiev fb650a2cc3 Merge pull request #503 from efiop/master 
cache: fix upper limit for sub dirs
 2018-03-02 15:10:41 +02:00
Ruslan Kuprieiev 488b49a852 cache: fix upper limit for sub dirs 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-02 15:03:49 +02:00
Ruslan Kuprieiev c423090bf5 Merge pull request #501 from efiop/master 
dvc: cloud: reuse connections when processing targets
 2018-03-02 04:14:43 +02:00
Ruslan Kuprieiev 8b85d1c201 dvc: cloud: reuse connections when processing targets 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-02 04:00:40 +02:00
Ruslan Kuprieiev 4d057575ef Merge pull request #500 from efiop/master 
test: fix data cloud test
 2018-03-02 03:34:03 +02:00