Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-09 15:28:26 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,494 Commits 7 Branches 579 Tags
1bcd7408ec3b09f78bfb540dd48dc3609459e4e4
Commit Graph
260 Commits
Author SHA1 Message Date
Clemens Wolff 6882a781a4 Add Azure Blob Storage remote 
This change adds a new cloud remote based on Azure Blob Storage [1] and
the azure-storage-blob SDK [2].

The implementation closely follows the patterns laid out by the existing
cloud remotes (AWS and GCP). Notable differences include:

- No use of a lock in the progress-bar callback and no requirement to
  look up total transfer size ahead of time: the Azure Storage SDK
  provides the total transfer bytes and current transfer offset in its
  progress callback.

- When `exists` is called, check the existence of every blob instead of
  listing the entire container. Container list operations can be
  expensive so unless the number of checked files is very large, making
  separate requests is likely going to be more efficient.

- For increased flexibility, access credentials for the Azure Storage
  backend can be provided either connection-string style via the remote
  URL or via environment variables. The Azure Storage backend can be
  emulated via Azurite [3].

[1] https://azure.microsoft.com/en-us/services/storage/blobs/
[2] https://github.com/Azure/azure-storage-python
[3] https://github.com/Azure/azurite

This change was created together with @EricSchles.
 2018-07-09 14:47:40 -04:00
Ruslan Kuprieiev b952e599bf debug: add a few more sleep()s to test_repro 2018-07-03 02:43:01 +03:00
Ruslan Kuprieiev 71d03b5a43 test: add sleep() calls 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-03 01:33:13 +03:00
Ruslan Kuprieiev f471f7714b dvc: uptimize add/checkout/state 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-03 00:33:05 +03:00
Ruslan Kuprieiev d37828c4e3 state: save collected dir to cache 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-02 20:41:35 +03:00
Ruslan Kuprieiev 67e0c24ced tests: add test_updater 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 23:29:07 +03:00
Ruslan Kuprieiev f634bc1916 test: remote: add basic test for remove() and checkout() 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 22:34:48 +03:00
Ruslan Kuprieiev f274a745a0 tests: checkout: test CLI for single files 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 21:02:43 +03:00
Ruslan Kuprieiev e8818ef67a test: add basic test from dvc import cmd 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 21:02:43 +03:00
Ruslan Kuprieiev 449d4880d1 tests: add test for double install 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 21:02:43 +03:00
Ruslan Kuprieiev 7c90cbab79 test: test root 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 21:02:43 +03:00
Ruslan Kuprieiev bd254c15ab test: remove updater test 2018-07-01 20:38:10 +03:00
Ruslan Kuprieiev 8d662e4402 test: del CI env when testing updater 2018-07-01 20:33:19 +03:00
Ruslan Kuprieiev 6219b28be6 test: add test for Updater 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 20:13:27 +03:00
Ruslan Kuprieiev 95f3b49db8 test: add basic test for metrics 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 20:07:00 +03:00
Ruslan Kuprieiev a392289cbf travis: run hdfs test only on Linux 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 10:02:43 +03:00
Ruslan Kuprieiev 591cd90583 travis: enable hdfs test 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 09:44:42 +03:00
Ruslan Kuprieiev fa9a79008b test: use gauth 2018-07-01 03:01:16 +03:00
Ruslan Kuprieiev 432fa56315 test: don't set gs env when running a command 2018-07-01 02:58:30 +03:00
Ruslan Kuprieiev 24d44695f7 test: fix typo 2018-07-01 02:45:02 +03:00
Ruslan Kuprieiev fde11a3b7f test: use GOOGLE_APPLICATION_CREDENTIALS when running GS repro test 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 02:36:22 +03:00
Ruslan Kuprieiev fb4fc72642 tests: try enable GS test on CI 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 02:08:06 +03:00
Ruslan Kuprieiev 12edcbaa04 remote: support HDFS for push/pull/status from local repo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-30 22:54:49 +03:00
Ruslan Kuprieiev 5d85b19e77 test: enable ssh test on osx 
Fixes #798

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-30 16:56:17 +03:00
Ruslan Kuprieiev 555e802ed2 RFC/perf: data cloud logic 
Properly splitting jobs between thread workers and removing unnecessary logic.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-30 07:44:11 +03:00
Ruslan Kuprieiev 50578a5923 gc: support s3/gs/ssh/hdfs 
Fixes #738

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-28 21:32:57 +03:00
Ruslan Kuprieiev add9498e82 gc: move gc logic to Remote 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-28 18:05:55 +03:00
Ruslan Kuprieiev 3f63592c82 test: two repositories one cache dir 
Fixes #699

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-27 21:35:02 +03:00
Ruslan Kuprieiev b8371d1c76 dvc: bring back dvc lock with dvc unlock 
Fixes #523

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-26 02:53:07 +03:00
Ruslan Kuprieiev 38aed70995 dvc: introduce dvc move 
Fixes #336

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-24 23:28:22 +03:00
Ruslan Kuprieiev 0b29f8748f test: use contents without \n 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-24 18:21:20 +03:00
Ruslan Kuprieiev 71752f68a8 hdfs: fix drivers and tests 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-24 17:23:47 +03:00
Ruslan Kuprieiev db7251b3ea dvc: support ssh deps/outs/cache 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-23 18:47:52 +03:00
Ruslan Kuprieiev c274bfee3e RFC: remove unused code 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-22 18:46:09 +03:00
Ruslan Kuprieiev d0b730f62f RFC: data_cloud and remotes 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-22 18:21:50 +03:00
Ruslan Kuprieiev 0a951f5ab3 dvc: move cloud/ drivers to remote/ 
Fixes #741

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-20 19:42:33 +03:00
Ruslan Kuprieiev 11f7eb0c4e system: add temporary workaround for getdirinfo() issue 
Fixes #670

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-17 19:14:58 +03:00
Ruslan Kuprieiev 9680e040a3 test: run tests in NCORES processes 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-17 16:04:52 +03:00
Ruslan Kuprieiev 834f9502dd test: enable CLI metrics test on windows 
Fixes #674

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-17 15:39:10 +03:00
Ruslan Kuprieiev 6886fc29d5 dvc: support dependecy files under tracked directory 
Fixes #722

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-17 09:32:29 +03:00
Ruslan Kuprieiev 56ed60f1cd dvc: add -d|--default option for dvc remote add 
Fixes #783

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-17 08:11:07 +03:00
Ruslan Kuprieiev c733b8130b dvc: add dvc metrics add/show/remove and dvc run -M|--metrics-no-cache 
Fixes #656

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-16 16:01:32 +03:00
Ruslan Kuprieiev 4271ddf11c dvc introduce dvc import 
Fixes #705

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-15 18:37:50 +03:00
Ruslan Kuprieiev b074c49318 dvc: support external local deps 
Fixes #764

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-15 03:35:53 +03:00
Ruslan Kuprieiev b57f829b1b utils: read in binary mode when converting CRLF -> LF 
Fixes #773

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-14 23:37:10 +03:00
Ruslan Kuprieiev 78e5b166fe dvc: replace StageNotFoundError with NotDvcFileError 
Fixes #760

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-13 14:00:03 +03:00
Ruslan Kuprieiev 5902e6066c remove: add -o|--outs and -p|--purge options 
Fixes #761

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-13 13:15:40 +03:00
Ruslan Kuprieiev ff0d91737b dvc: initial hdfs support 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-10 05:32:16 +03:00
Ruslan Kuprieiev a734e07776 dvc: support deps/outputs that are relative to specific remote 
Fixes #748

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-09 02:03:02 +03:00
Ruslan Kuprieiev c136297b82 stage: try to specify user's shell when running commands 
Fixes #749

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-07 23:19:29 +03:00