Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-10 15:28:34 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,875 Commits 7 Branches 579 Tags
23d00a15b9ca2e65c19ff63e72b1860b884198bb
Commit Graph
53 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 9135944d7c reqs: use grandalf 0.6 2018-08-23 04:41:49 +03:00
Ruslan Kuprieiev c378e4b3d2 dvc: pipeline show: create ASCII visualizations for DAG 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-08-19 21:01:12 +03:00
Ruslan Kuprieiev f02bf73587 reqs: only requre futures pkg on python 2.7 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-08-03 20:55:23 +03:00
Ruslan Kuprieiev d0ce57adac reqs: move pytest to test-requirements 
Fixes #966

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-08-03 13:33:24 +03:00
Ruslan Kuprieiev eaa994527d reqs: don't forget futures 
Fixes #967

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-08-03 13:25:36 +03:00
Ruslan Kuprieiev f1a28c3812 dvc: support dvc add for external files 
Fixes #890

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-16 16:34:16 +03:00
Clemens Wolff 6882a781a4 Add Azure Blob Storage remote 
This change adds a new cloud remote based on Azure Blob Storage [1] and
the azure-storage-blob SDK [2].

The implementation closely follows the patterns laid out by the existing
cloud remotes (AWS and GCP). Notable differences include:

- No use of a lock in the progress-bar callback and no requirement to
  look up total transfer size ahead of time: the Azure Storage SDK
  provides the total transfer bytes and current transfer offset in its
  progress callback.

- When `exists` is called, check the existence of every blob instead of
  listing the entire container. Container list operations can be
  expensive so unless the number of checked files is very large, making
  separate requests is likely going to be more efficient.

- For increased flexibility, access credentials for the Azure Storage
  backend can be provided either connection-string style via the remote
  URL or via environment variables. The Azure Storage backend can be
  emulated via Azurite [3].

[1] https://azure.microsoft.com/en-us/services/storage/blobs/
[2] https://github.com/Azure/azure-storage-python
[3] https://github.com/Azure/azurite

This change was created together with @EricSchles.
 2018-07-09 14:47:40 -04:00
Ruslan Kuprieiev 5d3a009886 metrics: require ply>=3.9 to avoid bug on binary windows package 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-08 01:02:26 +03:00
Ruslan Kuprieiev fba1d8c181 dvc: use heuristics to detect binary files 
Fixes #859

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-06 01:32:14 +03:00
Ruslan Kuprieiev 82890ec2a4 reqs: lock google-cloud==0.32.0 
0.33.0 is broken.
 2018-05-22 23:54:59 +03:00
Ruslan Kuprieiev 7709f57f0b reqs: add pytest(required by setuptools on some systems) 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-17 18:30:15 +03:00
Ruslan Kuprieiev 0f8f682a6b build: remove uneeded deps of boto 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-17 16:40:27 +03:00
Ruslan Kuprieiev 91882bf8c4 reqs: freeze google-compute-engine at 2.4.1 2018-05-17 15:15:38 +03:00
Ruslan Kuprieiev d9a2cbb3ce reqs: add wheel to requirements.txt 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-17 14:04:11 +03:00
Ruslan Kuprieiev 76ed66e1af reqs: require PyInstaller 3.3.1 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-17 13:09:48 +03:00
Ruslan Kuprieiev 03ebdb5cf2 dvc: use our own api instead of github's for version retrieval 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-10 14:40:59 +03:00
Ruslan Kuprieiev 6e20e1f845 dvc: check updates and warn about using an older version 
Fixes #687

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-05-06 22:12:28 +03:00
Ruslan Kuprieiev 0a049c232b cache: configurable cache dir and cache type 
From now on we allow specifying standalone cache dirs as well as
specifying linking method(copy, symlink, hardlink, reflink) for it.
If 'cache type' is not specified in the config, dvc will try all of
them starting from reflink all the way down to copy.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-28 21:50:49 +03:00
Ruslan Kuprieiev 832af146c6 requirements: add hidden requirement 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-19 21:03:07 +03:00
Ruslan Kuprieiev 1e7d548ca4 dvc: add dvc metrics 
Fixes #592

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-19 20:47:37 +03:00
Ruslan Kuprieiev f30e6e53c5 utils: convert to unix format when calculating md5 
Fixes #619

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-14 22:07:29 +03:00
Ruslan Kuprieiev 3748164ddb cloud: aws: migrate to boto3 
This allows us to abandon our own additional config parameters for
aws cloud(like region) and use the ones from .aws/config by default.
Plus, boto3 automatically switches to multipartupload if file is bigger
than a certain limit, allowing us to remove a lot of our own code.

Fixes #650
Fixes #636
Fixes #601

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-13 06:06:52 +03:00
Ruslan Kuprieiev 8839e72f07 dvc: stop using checksumdir and use our own way of checksuming a dir 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-10 01:56:06 +03:00
Ruslan Kuprieiev bd6688ff57 cloud: add ssh driver 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-04-08 22:26:18 +03:00
Ruslan Kuprieiev 13e0e9d2e4 dvc: use getdirinfo from ntfsutils 
We can now use getdirinfo directly, since our patch
https://github.com/sid0/ntfs/pull/10 is now merged
and released in ntfsutils 0.1.4.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-22 13:27:08 +02:00
Ruslan Kuprieiev 6b285e611c dvc: validate stage files 
Fixes #578

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-20 10:40:37 +02:00
Ruslan Kuprieiev e67be7afca dvc: add pyasn1 to requirements 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-18 13:11:44 +02:00
Ruslan Kuprieiev fa22e56917 state: use str representations for inode and mtime 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-18 12:35:56 +02:00
Ruslan Kuprieiev b7e9c9f021 pip: add setuptools version dependency 
Fixes #555

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-16 12:11:22 +02:00
Ruslan Kuprieiev b728d942c7 state: optimize state storage and computation 
Reduces 'dvc add' time for 30K files from 25sec to 9sec.

Fixes #546

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-03-13 20:18:03 +02:00
Ruslan Kuprieiev 96bb4640a3 Remove pygraphviz from requirements.txt 2018-03-12 18:58:15 +02:00
Ruslan Kuprieiev 068f742852 dvc: use zc.lockfile instead of fasteners 
fasteners' InterProcessLock doesn't work within
one process, thus it is not suitable for our dvc
library.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-02-27 04:20:32 +02:00
Ruslan Kuprieiev 4e2d6d1cd1 dvc: cleanup requirements 
There are a lot of old unused dependecies that need to be cleaned up.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-02-24 02:29:03 +02:00
Ruslan Kuprieiev 385317eb4c dvc: relax version requirements 
Fixes #480

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-02-24 02:29:03 +02:00
Ruslan Kuprieiev e4d9755a7e dvc: cloud RFC add support for directories 
Fixes #367

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-01-22 03:19:29 +02:00
Ruslan Kuprieiev 3b09290ed8 test: improve coverage and fix bugs 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-01-10 19:39:41 +02:00
Ruslan Kuprieiev 9c10a643a7 win: enable unit tests 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-01-02 13:03:06 +02:00
Ruslan Kuprieiev a779424a36 WIP RFC 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-12-23 09:37:40 +02:00
Ruslan Kuprieiev a828fee479 requirements: add pyyaml 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-11-28 21:47:23 +02:00
Dmitry 8b489f7663 Merge changes 2017-09-16 23:13:16 -07:00
Dmitry 82b51efe19 Add xmltodict to requirements for functests 2017-09-16 22:44:56 -07:00
Ruslan Kuprieiev 0c561e7f12 google cloud: fix cloud settings and requirements 
These are to allow using OAuth with your dvc, as
this is the recomended way to access gcp.

Fixes #203

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-09-14 22:23:22 +03:00
Ruslan Kuprieiev b707dbd0fe dvc: add 'dvc visual' 
This command generates dependency graph for specified target,
saves it in .dot file and also draws it into .svg file.

Fixes #39

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-08-26 09:18:45 +03:00
Ruslan Kuprieiev 194e1ec419 dvc: add 'config' command 
Fixes #177

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-08-20 06:19:38 +03:00
Ruslan Kuprieiev 7a3b500390 dvc: bring back frozen versions for deps 
Fixes #173.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-08-16 19:39:05 +03:00
Ruslan Kuprieiev 883e6773f7 dvc: get rid of pathlib 
Fixes #118

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-08-03 23:31:59 +03:00
Ruslan Kuprieiev b1f19cee37 travis: build linux packages 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-07-31 03:45:10 +03:00
Ruslan Kuprieiev e02437ed97 dvc: logger: use colorama to colorize output 
Colors help readability, especially in verbose mode.
Current color map:

	Debug:   Blue
	Info:    White
	Warning: Yellow
	Error:   Red
 2017-05-22 14:44:31 +03:00
Ruslan Kuprieiev ba216474f1 requirements: move mac-specific to osx.txt 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2017-05-02 09:18:29 +03:00
Earl Hathaway 0a6d831bfc tests around reading aws credentials 2017-04-22 17:28:02 -07:00