It is generally recommended to use the subprocess module instead of the
os module to run subshell commands given since the former makes it
easier to comunicate with the process, has an improved security model,
etc. More information is available in [1].
Note that this change uses `check_output` instead of `check_call` to
prevent the command output from being printed to stdout.
This change also introduces two additional enhancements:
- Skip the hadoop test if the hadoop binary is not available.
- Skip the ssh test if the ssh binary is not available.
[1] https://docs.python.org/3/library/subprocess.html#replacing-os-system
This change adds a new cloud remote based on Azure Blob Storage [1] and
the azure-storage-blob SDK [2].
The implementation closely follows the patterns laid out by the existing
cloud remotes (AWS and GCP). Notable differences include:
- No use of a lock in the progress-bar callback and no requirement to
look up total transfer size ahead of time: the Azure Storage SDK
provides the total transfer bytes and current transfer offset in its
progress callback.
- When `exists` is called, check the existence of every blob instead of
listing the entire container. Container list operations can be
expensive so unless the number of checked files is very large, making
separate requests is likely going to be more efficient.
- For increased flexibility, access credentials for the Azure Storage
backend can be provided either connection-string style via the remote
URL or via environment variables. The Azure Storage backend can be
emulated via Azurite [3].
[1] https://azure.microsoft.com/en-us/services/storage/blobs/
[2] https://github.com/Azure/azure-storage-python
[3] https://github.com/Azure/azurite
This change was created together with @EricSchles.
From now on we allow specifying standalone cache dirs as well as
specifying linking method(copy, symlink, hardlink, reflink) for it.
If 'cache type' is not specified in the config, dvc will try all of
them starting from reflink all the way down to copy.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This allows us to abandon our own additional config parameters for
aws cloud(like region) and use the ones from .aws/config by default.
Plus, boto3 automatically switches to multipartupload if file is bigger
than a certain limit, allowing us to remove a lot of our own code.
Fixes#650Fixes#636Fixes#601
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Since we can now detect cache corruption, we don't need
to force any permissions on our data files.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev
Ruslan Kuprieiev
Clemens Wolff