remote: add support for aliyun oss (#1961)

* remote: add support for aliyun oss

Usage:
$ dvc remote add myremote oss://my-bucket.endpoint/path
Set key id and key secret using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"

Ref:
oss python SDK: https://www.alibabacloud.com/help/doc-detail/32026.htm

* Not needed, since we don't support external dependencies/outputs on OSS.
See https://github.com/iterative/dvc/pull/1961.

* Add a way to test oss storage using docker.

Start a container running an oss emulator.
$ git clone https://github.com/nanaya-tachibana/oss-emulator.git
$ docker image build -t oss:1.0 oss-emulator
$ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0
Setup environment variables.
$ export OSS_BUCKET='my-bucket'
$ export OSS_ENDPOINT='localhost:8880'
$ export OSS_ACCESS_KEY_ID='AccessKeyID'
$ export OSS_ACCESS_KEY_SECRET='AccessKeySecret'

* Use default key id and key secret when they are not given,
which gives read access to public read bucket and public bucket.

* test: add oss tests to appveyor.

* remove unneeded tests.

* remote: use s3 style url for oss storage and make endpoint a configurable value.

Usage:
$ dvc remote add myremote oss://my-bucket/path
Set key id, key secret and endpoint using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
$ dvc remote modify myremote oss_endpoint endpoint
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"
$ export OSS_ENDPOINT="endpoint"

* remote: fallback to [] if there are no cinfos

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: remote: add oss CLI test

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* travis: use iterative's fork of oss-emulator

Just to keep things in-house.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
This commit is contained in:
nanaya-tachibana
2019-05-09 15:35:41 +03:00
committed by Ruslan Kuprieiev
parent 6408b58b8d
commit 7ceaf88c54
13 changed files with 310 additions and 7 deletions
+4 -4
View File
@@ -24,7 +24,7 @@
.. image:: https://codecov.io/gh/iterative/dvc/branch/master/graph/badge.svg
:target: https://codecov.io/gh/iterative/dvc
:alt: Codecov
.. image:: https://img.shields.io/badge/patreon-donate-green.svg
:target: https://www.patreon.com/DVCorg/overview
:alt: Donate
@@ -38,7 +38,7 @@ machine learning projects. Key features:
any databases. Does not depend on any proprietary online services;
#. it manages and versions **datasets** and **machine learning models**. Data is saved in
S3, Google cloud, Azure, SSH server, HDFS or even local HDD RAID;
S3, Google cloud, Azure, Alibaba cloud, SSH server, HDFS or even local HDD RAID;
#. it makes projects **reproducible** and **shareable**, it helps answering question how
the model was build;
@@ -63,7 +63,7 @@ made right and tailored specifically for ML and Data Science scenarios.
#. ``Git/Git-lfs`` part - DVC helps you storing and sharing data artifacts, models. It connects them with your
Git repository.
#. ``Makefiles`` part - DVC describes how one data or model artifact was build from another data.
#. ``Makefiles`` part - DVC describes how one data or model artifact was build from another data.
DVC usually runs along with Git. Git is used as usual to store and version code and DVC meta-files. DVC helps
to store data and model files seamlessly out of Git while preserving almost the same user experience as if they
@@ -118,7 +118,7 @@ pip (PyPI)
pip install dvc
Depending on the remote storage type you plan to use to keep and share your data, you might need to specify
one of the optional dependencies: ``s3``, ``gs``, ``azure``, ``ssh``. Or ``all_remotes`` to include them all.
one of the optional dependencies: ``s3``, ``gs``, ``azure``, ``oss``, ``ssh``. Or ``all_remotes`` to include them all.
The command should look like this: ``pip install dvc[s3]`` - it installs the ``boto3`` library along with
DVC to support the AWS S3 storage.