* dvc: dedup used cache
This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts
* dvc: delay checksum names concatenation
* wip - pull is working
* use fetch instead of reproduce
* test pull and fetch
* style changes
* fetch: catch config error on fetch instead of get_remote
* output: remove unused guard clause
* fetch: define a more specific error instead of ConfigError
* fetch: display number of errors correctly
* exceptions: remove "Failed" prefix to download/upload
* fetch: catch other exceptions that might halt the download
* fetch: log catched exceptions
* fetch: reformat exception
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
* fix formatting
As a part of this:
- DataCloud._cloud was refactored into `.get_remote()`
- data stopped using internal _cloud
- added a parametrization for remote tests
* remote: add support for aliyun oss
Usage:
$ dvc remote add myremote oss://my-bucket.endpoint/path
Set key id and key secret using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"
Ref:
oss python SDK: https://www.alibabacloud.com/help/doc-detail/32026.htm
* Not needed, since we don't support external dependencies/outputs on OSS.
See https://github.com/iterative/dvc/pull/1961.
* Add a way to test oss storage using docker.
Start a container running an oss emulator.
$ git clone https://github.com/nanaya-tachibana/oss-emulator.git
$ docker image build -t oss:1.0 oss-emulator
$ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0
Setup environment variables.
$ export OSS_BUCKET='my-bucket'
$ export OSS_ENDPOINT='localhost:8880'
$ export OSS_ACCESS_KEY_ID='AccessKeyID'
$ export OSS_ACCESS_KEY_SECRET='AccessKeySecret'
* Use default key id and key secret when they are not given,
which gives read access to public read bucket and public bucket.
* test: add oss tests to appveyor.
* remove unneeded tests.
* remote: use s3 style url for oss storage and make endpoint a configurable value.
Usage:
$ dvc remote add myremote oss://my-bucket/path
Set key id, key secret and endpoint using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
$ dvc remote modify myremote oss_endpoint endpoint
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"
$ export OSS_ENDPOINT="endpoint"
* remote: fallback to [] if there are no cinfos
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* test: remote: add oss CLI test
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* travis: use iterative's fork of oss-emulator
Just to keep things in-house.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
- Uses a formatter for coloring output
- Move the `progress_aware` decorator logic to the logger itself
- Refactor modules so each one uses their own instance of a logger
Close#1753
1) split Project's methods into separate files;
2) rename Project into Repo(similar to gitpython's Repo)
Signed-off-by: Ruslan Kuprieiev <[email protected]>
In scenarios with millions of files, late shorcutting of
debug msgs results in massive overhead. In order to avoid
that, just shortcut logger messages ASAP if verbose mode
is not enabled.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This change adds a new cloud remote based on Azure Blob Storage [1] and
the azure-storage-blob SDK [2].
The implementation closely follows the patterns laid out by the existing
cloud remotes (AWS and GCP). Notable differences include:
- No use of a lock in the progress-bar callback and no requirement to
look up total transfer size ahead of time: the Azure Storage SDK
provides the total transfer bytes and current transfer offset in its
progress callback.
- When `exists` is called, check the existence of every blob instead of
listing the entire container. Container list operations can be
expensive so unless the number of checked files is very large, making
separate requests is likely going to be more efficient.
- For increased flexibility, access credentials for the Azure Storage
backend can be provided either connection-string style via the remote
URL or via environment variables. The Azure Storage backend can be
emulated via Azurite [3].
[1] https://azure.microsoft.com/en-us/services/storage/blobs/
[2] https://github.com/Azure/azure-storage-python
[3] https://github.com/Azure/azurite
This change was created together with @EricSchles.
From now on, we identify our cache files by their md5sum,
so that we can avoid cache collission in branching scenario
and also dedup same data at the same time.
Fixes#250, #251
Signed-off-by: Ruslan Kuprieiev <[email protected]>
These are to allow using OAuth with your dvc, as
this is the recomended way to access gcp.
Fixes#203
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Saugat Pachhai
Ruslan Kuprieiev
Alexander Schepanovski
Mr. OutisandRuslan Kuprieiev
pawel
nanaya-tachibana
Clemens Wolff
Dmitry Petrov