Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-07 15:28:28 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,504 Commits 7 Branches 579 Tags
fcbc6d115cba99cfa60aed55b6d6669bcacecc07
Commit Graph
1504 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Clemens Wolff fcbc6d115c Wire-up Azure in TestDataCloud 2018-07-10 16:33:52 -04:00
Clemens Wolff 1b1ca68b64 Fix inspections 2018-07-10 16:33:52 -04:00
Clemens Wolff 9724bd94aa Replace os.system with subprocess.check_output 
It is generally recommended to use the subprocess module instead of the
os module to run subshell commands given since the former makes it
easier to comunicate with the process, has an improved security model,
etc. More information is available in [1].

Note that this change uses `check_output` instead of `check_call` to
prevent the command output from being printed to stdout.

This change also introduces two additional enhancements:
- Skip the hadoop test if the hadoop binary is not available.
- Skip the ssh test if the ssh binary is not available.

[1] https://docs.python.org/3/library/subprocess.html#replacing-os-system
 2018-07-10 16:33:14 -04:00
Clemens Wolff 581132bbe0 Fix environment variable name 2018-07-10 13:54:49 -04:00
Ruslan Kuprieiev e051964a1b Merge pull request #873 from efiop/master 
collect: skip if cache dir doesn't exist
 2018-07-10 20:09:44 +03:00
Ruslan Kuprieiev 2aa9288f55 collect: skip if cache dir doesn't exist 
Related to #872

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-10 19:22:18 +03:00
Ruslan Kuprieiev 6996d0a9e6 Merge pull request #870 from efiop/master 
remote: add comments for s3/azure exists() method
 2018-07-09 23:49:48 +03:00
Ruslan Kuprieiev 1f30d29b57 remote: add comments for s3/azure exists() method 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-09 23:40:56 +03:00
Ruslan Kuprieiev abee5f07bb Merge pull request #869 from CatalystCode/code-climate 
Fix CodeClimate inspections
 2018-07-09 23:21:02 +03:00
Clemens Wolff afd46e1a8a Fix CodeClimate inspections 2018-07-09 16:06:10 -04:00
Ruslan Kuprieiev 1bcd7408ec Merge pull request #868 from CatalystCode/azure-blob-storage-remote 
Add Azure Blob Storage remote
 2018-07-09 23:01:07 +03:00
Clemens Wolff 6882a781a4 Add Azure Blob Storage remote 
This change adds a new cloud remote based on Azure Blob Storage [1] and
the azure-storage-blob SDK [2].

The implementation closely follows the patterns laid out by the existing
cloud remotes (AWS and GCP). Notable differences include:

- No use of a lock in the progress-bar callback and no requirement to
  look up total transfer size ahead of time: the Azure Storage SDK
  provides the total transfer bytes and current transfer offset in its
  progress callback.

- When `exists` is called, check the existence of every blob instead of
  listing the entire container. Container list operations can be
  expensive so unless the number of checked files is very large, making
  separate requests is likely going to be more efficient.

- For increased flexibility, access credentials for the Azure Storage
  backend can be provided either connection-string style via the remote
  URL or via environment variables. The Azure Storage backend can be
  emulated via Azurite [3].

[1] https://azure.microsoft.com/en-us/services/storage/blobs/
[2] https://github.com/Azure/azure-storage-python
[3] https://github.com/Azure/azurite

This change was created together with @EricSchles.
 2018-07-09 14:47:40 -04:00
Ruslan Kuprieiev 50ea36e740 dvc: bump to 0.10.2 0.10.2 2018-07-08 01:13:21 +03:00
Ruslan Kuprieiev c7ed95f5ba Merge pull request #867 from efiop/master 
metrics bugfixes
 2018-07-08 01:11:03 +03:00
Ruslan Kuprieiev 5d3a009886 metrics: require ply>=3.9 to avoid bug on binary windows package 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-08 01:02:26 +03:00
Ruslan Kuprieiev 6683f13b4e metric: print error if unable to read 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-08 01:02:26 +03:00
Ruslan Kuprieiev 3f73ef057a dvc: bump to 0.10.1 0.10.1 2018-07-07 03:43:16 +03:00
Ruslan Kuprieiev 961bf8b34c Merge pull request #866 from efiop/master 
metrics: show: don't print branch if all_branches=False
 2018-07-07 03:41:39 +03:00
Ruslan Kuprieiev 11de88fcd4 metrics: show: don't print branch if all_branches=False 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-07 03:31:42 +03:00
Ivan Shcheklein e83dd1c972 fix README tutorial link 2018-07-05 20:51:25 -07:00
Ruslan Kuprieiev 62cd68af77 Merge pull request #864 from efiop/master 
updater: don't forget to convert to int
 2018-07-06 06:31:45 +03:00
Ruslan Kuprieiev 715cc717aa updater: don't forget to convert to int 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-06 05:30:14 +03:00
Ruslan Kuprieiev d5c015ff63 dvc: bump to 0.10.0 0.10.0 2018-07-06 04:56:30 +03:00
Ruslan Kuprieiev db374b6553 Merge pull request #863 from efiop/master 
brancher: don't checkout if no branches is specified
 2018-07-06 04:12:05 +03:00
Ruslan Kuprieiev 01efbac5a3 brancher: don't checkout if no branches is specified 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-06 04:05:27 +03:00
Ruslan Kuprieiev 0f940f9167 appveyor: pull images before deploy 2018-07-06 03:39:47 +03:00
Ruslan Kuprieiev e85368c8b8 Merge pull request #862 from efiop/master 
dvc: use heuristics to detect binary files
 2018-07-06 01:39:13 +03:00
Ruslan Kuprieiev fba1d8c181 dvc: use heuristics to detect binary files 
Fixes #859

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-06 01:32:14 +03:00
Ruslan Kuprieiev 6e82c6c748 Merge pull request #861 from efiop/master 
dvc: properly save symlinks
 2018-07-05 20:41:24 +03:00
Ruslan Kuprieiev 468f4beee7 dvc: properly save symlinks 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-05 20:00:58 +03:00
Ruslan Kuprieiev 08b3273fcc Merge pull request #860 from efiop/master 
dvc: don't forget to add state and state.lock to gitignore
 2018-07-05 16:59:24 +03:00
Ruslan Kuprieiev b7b07d393a dvc: don't forget to add state and state.lock to gitignore 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-05 16:48:23 +03:00
Ivan Shcheklein 2542ceca0d update README, documenation links, flow image 2018-07-04 23:11:49 -07:00
Ruslan Kuprieiev 9b032b72cb Merge pull request #858 from efiop/master 
docs: remove from the core repository
 2018-07-04 20:31:16 +03:00
Ruslan Kuprieiev 0a173c4731 docs: remove from the core repository 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-04 20:23:20 +03:00
Ruslan Kuprieiev 65b5dcb601 Merge pull request #854 from efiop/master 
link: properly iterate cache types
 2018-07-03 03:25:24 +03:00
Ruslan Kuprieiev 425e44bac8 link: properly iterate cache types 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-03 03:12:32 +03:00
Ruslan Kuprieiev b952e599bf debug: add a few more sleep()s to test_repro 2018-07-03 02:43:01 +03:00
Ruslan Kuprieiev adc34eb312 Merge pull request #851 from efiop/master 
Misc fixes
 2018-07-03 01:41:05 +03:00
Ruslan Kuprieiev 71d03b5a43 test: add sleep() calls 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-03 01:33:13 +03:00
Ruslan Kuprieiev db5cb125fb add: fix error msg 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-03 01:23:03 +03:00
Ruslan Kuprieiev 45fc17a074 Merge pull request #849 from efiop/master 
dvc: misc improvements
 2018-07-03 00:45:52 +03:00
Ruslan Kuprieiev e6bcdd3979 link: exclude unsupported link types after trying once 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-03 00:33:05 +03:00
Ruslan Kuprieiev f471f7714b dvc: uptimize add/checkout/state 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-03 00:33:05 +03:00
Ruslan Kuprieiev d37828c4e3 state: save collected dir to cache 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-02 20:41:35 +03:00
Ruslan Kuprieiev d9b1308447 state: store collected dir info 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-02 19:54:17 +03:00
Ruslan Kuprieiev ecf8045887 state: store cache state on per-repo basis 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-02 19:04:40 +03:00
Ruslan Kuprieiev 8d6813a112 checkout: use link_state properly 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-02 18:54:33 +03:00
Ruslan Kuprieiev 35d318f0f0 Merge pull request #847 from yustoris/feature/add-aws-remote-url 
RFP: Add endpoint url config for AWS
 2018-07-02 16:15:30 +03:00
kenji.yamauchi 0f8288fd9a Add endpoint url config for AWS 2018-07-02 12:44:33 +09:00