* Add info for link and filesystem type
* Remove unnecessary `repo_dir` fixture
* Add cache info
* Classify `test_version.py` as functional test
* Remove redundancy
* Optimize some functions involved
* Add pathlib2 to install_requires
* Display fs type for cache directory
* Use `dvc.utils.compat` to import Path
* Remove redundancy in tests and make logs more descriptive
* Coalesce `requirements.txt` and `tests/requirements.txt` into `setup.py`
* Remove requirements for tests
* Parse tests requirements in setup.py
* Append extra dependenices in test_requirements in setup.py
* Modify CI build to pass the tests
* Correctly install black requirements
* Install all requirements to pass the tests
* remote: add support for aliyun oss
Usage:
$ dvc remote add myremote oss://my-bucket.endpoint/path
Set key id and key secret using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"
Ref:
oss python SDK: https://www.alibabacloud.com/help/doc-detail/32026.htm
* Not needed, since we don't support external dependencies/outputs on OSS.
See https://github.com/iterative/dvc/pull/1961.
* Add a way to test oss storage using docker.
Start a container running an oss emulator.
$ git clone https://github.com/nanaya-tachibana/oss-emulator.git
$ docker image build -t oss:1.0 oss-emulator
$ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0
Setup environment variables.
$ export OSS_BUCKET='my-bucket'
$ export OSS_ENDPOINT='localhost:8880'
$ export OSS_ACCESS_KEY_ID='AccessKeyID'
$ export OSS_ACCESS_KEY_SECRET='AccessKeySecret'
* Use default key id and key secret when they are not given,
which gives read access to public read bucket and public bucket.
* test: add oss tests to appveyor.
* remove unneeded tests.
* remote: use s3 style url for oss storage and make endpoint a configurable value.
Usage:
$ dvc remote add myremote oss://my-bucket/path
Set key id, key secret and endpoint using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
$ dvc remote modify myremote oss_endpoint endpoint
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"
$ export OSS_ENDPOINT="endpoint"
* remote: fallback to [] if there are no cinfos
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* test: remote: add oss CLI test
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* travis: use iterative's fork of oss-emulator
Just to keep things in-house.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* test: remove never used copy of load_stage_file()
* dvc: save comments and custom formatting in stage files
* Add a couple of TODOs
* test: test repro doesn't clear comments and formatting in .dvc
As part of this I started writing pytest-like tests and added commonly
used fixtures.
* test: simplify and prettify dvc formatting retained test
Also test that line comments after md5 fields are retained.
* test: fix tests
* dvc: fix stage and metrics opens not closed
* dvc: always use utf-8 in stage files
It used to encode in system default encoding. Not sure if PyYAML was
handling this silently or we were just lucky all the way.
* test: unit test apply_diff()
* test: remove TODOs and ASK
* dvc: polish dvc-formatting PR
* dvc: move stage load/dump utils to a new stage utils file
* dvc: clean up apply_diff()
* test: test stage md5 ignores comments
* test: remove pytests starting marker
- Use `version.py` to compute a dynamic version stored in `__version__`.
Version information was scattered across the source code
(`VERSION_BASE`, `VERSION`, `__version__`, and `version.py`).
- Read `__version__` from `setup.py` and **pin** it using custom
`build_py` command class.
This way, we can maintain consistancy between the build version
and the runtime one.
Also, this gives us the ability to run `setup.py` without depending on
on `dvc` itself.
Close#1824
* updater: Add custom update message depending on package manager
Close#938Close#1031
* Reduce cognitive complexity and avoid too many return functions
* Use a dictionary instead of if/else
* Take into account pkg installation with OSX
* updater: add platform detection
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This change adds a new cloud remote based on Azure Blob Storage [1] and
the azure-storage-blob SDK [2].
The implementation closely follows the patterns laid out by the existing
cloud remotes (AWS and GCP). Notable differences include:
- No use of a lock in the progress-bar callback and no requirement to
look up total transfer size ahead of time: the Azure Storage SDK
provides the total transfer bytes and current transfer offset in its
progress callback.
- When `exists` is called, check the existence of every blob instead of
listing the entire container. Container list operations can be
expensive so unless the number of checked files is very large, making
separate requests is likely going to be more efficient.
- For increased flexibility, access credentials for the Azure Storage
backend can be provided either connection-string style via the remote
URL or via environment variables. The Azure Storage backend can be
emulated via Azurite [3].
[1] https://azure.microsoft.com/en-us/services/storage/blobs/
[2] https://github.com/Azure/azure-storage-python
[3] https://github.com/Azure/azurite
This change was created together with @EricSchles.
Andrew Grigorev
Ruslan Kuprieiev
Aman Sharma
Mr. Outis
nanaya-tachibana
Brett Barkley
Alexander Schepanovski
pared
Hugo
Zhanibek
Dmitrii Pukhov
Ivan Shcheklein
Ruslan Kuprieiev
Clemens Wolff
Victor Villas