This helps verify in the CI that the environment variables for the
integration tests are set up correctly since we can see exactly which
tests are being executed and which ones are being skipped.
Note: the tests are run against the Azurite [1] local emulator for Azure
Blob Storage and not against a live service.
[1] https://github.com/azure/azurite
It is generally recommended to use the subprocess module instead of the
os module to run subshell commands given since the former makes it
easier to comunicate with the process, has an improved security model,
etc. More information is available in [1].
Note that this change uses `check_output` instead of `check_call` to
prevent the command output from being printed to stdout.
This change also introduces two additional enhancements:
- Skip the hadoop test if the hadoop binary is not available.
- Skip the ssh test if the ssh binary is not available.
[1] https://docs.python.org/3/library/subprocess.html#replacing-os-system
This change adds a new cloud remote based on Azure Blob Storage [1] and
the azure-storage-blob SDK [2].
The implementation closely follows the patterns laid out by the existing
cloud remotes (AWS and GCP). Notable differences include:
- No use of a lock in the progress-bar callback and no requirement to
look up total transfer size ahead of time: the Azure Storage SDK
provides the total transfer bytes and current transfer offset in its
progress callback.
- When `exists` is called, check the existence of every blob instead of
listing the entire container. Container list operations can be
expensive so unless the number of checked files is very large, making
separate requests is likely going to be more efficient.
- For increased flexibility, access credentials for the Azure Storage
backend can be provided either connection-string style via the remote
URL or via environment variables. The Azure Storage backend can be
emulated via Azurite [3].
[1] https://azure.microsoft.com/en-us/services/storage/blobs/
[2] https://github.com/Azure/azure-storage-python
[3] https://github.com/Azure/azurite
This change was created together with @EricSchles.
Ruslan Kuprieiev
Ruslan Kuprieiev
Clemens Wolff
Ivan Shcheklein