mirror of
https://github.com/Stone-Red-Code/dvc.git
synced 2026-11-01 23:08:43 +01:00
remote: add support for aliyun oss (#1961)
* remote: add support for aliyun oss Usage: $ dvc remote add myremote oss://my-bucket.endpoint/path Set key id and key secret using modify command $ dvc remote modify myremote oss_key_id my-key-id $ dvc remote modify myremote oss_key_secret my-key-secret or environment variables $ export OSS_ACCESS_KEY_ID="my-key-id" $ export OSS_ACCESS_KEY_SECRET="my-key-secret" Ref: oss python SDK: https://www.alibabacloud.com/help/doc-detail/32026.htm * Not needed, since we don't support external dependencies/outputs on OSS. See https://github.com/iterative/dvc/pull/1961. * Add a way to test oss storage using docker. Start a container running an oss emulator. $ git clone https://github.com/nanaya-tachibana/oss-emulator.git $ docker image build -t oss:1.0 oss-emulator $ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0 Setup environment variables. $ export OSS_BUCKET='my-bucket' $ export OSS_ENDPOINT='localhost:8880' $ export OSS_ACCESS_KEY_ID='AccessKeyID' $ export OSS_ACCESS_KEY_SECRET='AccessKeySecret' * Use default key id and key secret when they are not given, which gives read access to public read bucket and public bucket. * test: add oss tests to appveyor. * remove unneeded tests. * remote: use s3 style url for oss storage and make endpoint a configurable value. Usage: $ dvc remote add myremote oss://my-bucket/path Set key id, key secret and endpoint using modify command $ dvc remote modify myremote oss_key_id my-key-id $ dvc remote modify myremote oss_key_secret my-key-secret $ dvc remote modify myremote oss_endpoint endpoint or environment variables $ export OSS_ACCESS_KEY_ID="my-key-id" $ export OSS_ACCESS_KEY_SECRET="my-key-secret" $ export OSS_ENDPOINT="endpoint" * remote: fallback to [] if there are no cinfos Signed-off-by: Ruslan Kuprieiev <[email protected]> * test: remote: add oss CLI test Signed-off-by: Ruslan Kuprieiev <[email protected]> * travis: use iterative's fork of oss-emulator Just to keep things in-house. Signed-off-by: Ruslan Kuprieiev <[email protected]>
This commit is contained in:
committed by
Ruslan Kuprieiev
parent
6408b58b8d
commit
7ceaf88c54
+4
-4
@@ -24,7 +24,7 @@
|
||||
.. image:: https://codecov.io/gh/iterative/dvc/branch/master/graph/badge.svg
|
||||
:target: https://codecov.io/gh/iterative/dvc
|
||||
:alt: Codecov
|
||||
|
||||
|
||||
.. image:: https://img.shields.io/badge/patreon-donate-green.svg
|
||||
:target: https://www.patreon.com/DVCorg/overview
|
||||
:alt: Donate
|
||||
@@ -38,7 +38,7 @@ machine learning projects. Key features:
|
||||
any databases. Does not depend on any proprietary online services;
|
||||
|
||||
#. it manages and versions **datasets** and **machine learning models**. Data is saved in
|
||||
S3, Google cloud, Azure, SSH server, HDFS or even local HDD RAID;
|
||||
S3, Google cloud, Azure, Alibaba cloud, SSH server, HDFS or even local HDD RAID;
|
||||
|
||||
#. it makes projects **reproducible** and **shareable**, it helps answering question how
|
||||
the model was build;
|
||||
@@ -63,7 +63,7 @@ made right and tailored specifically for ML and Data Science scenarios.
|
||||
|
||||
#. ``Git/Git-lfs`` part - DVC helps you storing and sharing data artifacts, models. It connects them with your
|
||||
Git repository.
|
||||
#. ``Makefiles`` part - DVC describes how one data or model artifact was build from another data.
|
||||
#. ``Makefiles`` part - DVC describes how one data or model artifact was build from another data.
|
||||
|
||||
DVC usually runs along with Git. Git is used as usual to store and version code and DVC meta-files. DVC helps
|
||||
to store data and model files seamlessly out of Git while preserving almost the same user experience as if they
|
||||
@@ -118,7 +118,7 @@ pip (PyPI)
|
||||
pip install dvc
|
||||
|
||||
Depending on the remote storage type you plan to use to keep and share your data, you might need to specify
|
||||
one of the optional dependencies: ``s3``, ``gs``, ``azure``, ``ssh``. Or ``all_remotes`` to include them all.
|
||||
one of the optional dependencies: ``s3``, ``gs``, ``azure``, ``oss``, ``ssh``. Or ``all_remotes`` to include them all.
|
||||
The command should look like this: ``pip install dvc[s3]`` - it installs the ``boto3`` library along with
|
||||
DVC to support the AWS S3 storage.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user