This change adds a new cloud remote based on Azure Blob Storage [1] and
the azure-storage-blob SDK [2].
The implementation closely follows the patterns laid out by the existing
cloud remotes (AWS and GCP). Notable differences include:
- No use of a lock in the progress-bar callback and no requirement to
look up total transfer size ahead of time: the Azure Storage SDK
provides the total transfer bytes and current transfer offset in its
progress callback.
- When `exists` is called, check the existence of every blob instead of
listing the entire container. Container list operations can be
expensive so unless the number of checked files is very large, making
separate requests is likely going to be more efficient.
- For increased flexibility, access credentials for the Azure Storage
backend can be provided either connection-string style via the remote
URL or via environment variables. The Azure Storage backend can be
emulated via Azurite [3].
[1] https://azure.microsoft.com/en-us/services/storage/blobs/
[2] https://github.com/Azure/azure-storage-python
[3] https://github.com/Azure/azurite
This change was created together with @EricSchles.
From now on we allow specifying standalone cache dirs as well as
specifying linking method(copy, symlink, hardlink, reflink) for it.
If 'cache type' is not specified in the config, dvc will try all of
them starting from reflink all the way down to copy.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This allows us to abandon our own additional config parameters for
aws cloud(like region) and use the ones from .aws/config by default.
Plus, boto3 automatically switches to multipartupload if file is bigger
than a certain limit, allowing us to remove a lot of our own code.
Fixes#650Fixes#636Fixes#601
Signed-off-by: Ruslan Kuprieiev <[email protected]>
fasteners' InterProcessLock doesn't work within
one process, thus it is not suitable for our dvc
library.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
These are to allow using OAuth with your dvc, as
this is the recomended way to access gcp.
Fixes#203
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This command generates dependency graph for specified target,
saves it in .dot file and also draws it into .svg file.
Fixes#39
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This will make the structure of dvc more linear
making it easier to read. Plus, these changes
are needed in preparation for libdvc, as they
move all cmdline parsing to one common place.
This patch also refactors our settings system
also make it more linear and remove abstraction
levels that are not needed and only cause confusion.
There is a lot more to be done in preparation for
dvc lib and RFC overall, but this a first step
in that long journey.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Clemens Wolff
Ruslan Kuprieiev
Victor Villas
Dmitry
Ruslan Kuprieiev