Related to #1629
Speeds up HDFS tests significantly. E.g. ExternalHDFS test goes from 600
sec to 190sec. test_open_external[HDFS] from 160sec to 3sec.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Replaced Path* dataclasses with PathInfo/URLInfo/CloudURLInfo, which
know how parse, serialize, stringify and join themselves. They also
provide useful accessors like part of an url or pathlib.Path like
.parent or .name.
Along the way two families of bugs where fixed and should never occur in
the future:
- paths not matched as strings anymore, e.g. using str.startswith()
- Output.path is eliminated as ambiguous
Both families might had caused undefined surprising behavior when local
and remote paths clash.
* Document dvc.scm.git::Git::_get_diff_trees and shorten commit hashes in dvc.scm.git::Git::get_diff_trees
* diff: Changes to dvc.scm.git so `dvc diff` outputs short SHAs
- Docstring for Git class constructor
- Renames self.git for self.repo throughout module (involves several test changes)
- Better doc and logical refactor in Git._get_diff_trees
- `dvc diff` now outputs short SHA strings (Git._get_diff_trees) - hard coded to 7 length str
(involves some test changes in tests/func/test_diff.py)
For #1902
* Undoes "Renames self.git for self.repo throughout module" introduced in 3d2dc83.
* autocomplete: Brings scripts/completion/dvc.zsh up-to-date
- Reorganizes commands in alphabetical order;
- Updates the command dscriptions according to their current usage (-h) output;
- Adds comments for MISSING command entries
* Update scripts/completion/dvc.zsh
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
* Update scripts/completion/dvc.zsh
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
* Update scripts/completion/dvc.zsh
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
* Update scripts/completion/dvc.zsh
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
* autocomplete: Adds missing commands to Zsh script
* cache: updates slow link type warning message (and test)
in accordance with https://github.com/iterative/dvc.org/pull/331
* Fixes accidental docstring rewrap
per https://github.com/iterative/dvc/pull/2003#discussion_r286206870
* Actually fixes accidental docstring rewrap...
per https://github.com/iterative/dvc/pull/2003#pullrequestreview-240994592
* Really fixes accidental docstring rewrap this time...
* remote: add support for aliyun oss
Usage:
$ dvc remote add myremote oss://my-bucket.endpoint/path
Set key id and key secret using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"
Ref:
oss python SDK: https://www.alibabacloud.com/help/doc-detail/32026.htm
* Not needed, since we don't support external dependencies/outputs on OSS.
See https://github.com/iterative/dvc/pull/1961.
* Add a way to test oss storage using docker.
Start a container running an oss emulator.
$ git clone https://github.com/nanaya-tachibana/oss-emulator.git
$ docker image build -t oss:1.0 oss-emulator
$ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0
Setup environment variables.
$ export OSS_BUCKET='my-bucket'
$ export OSS_ENDPOINT='localhost:8880'
$ export OSS_ACCESS_KEY_ID='AccessKeyID'
$ export OSS_ACCESS_KEY_SECRET='AccessKeySecret'
* Use default key id and key secret when they are not given,
which gives read access to public read bucket and public bucket.
* test: add oss tests to appveyor.
* remove unneeded tests.
* remote: use s3 style url for oss storage and make endpoint a configurable value.
Usage:
$ dvc remote add myremote oss://my-bucket/path
Set key id, key secret and endpoint using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
$ dvc remote modify myremote oss_endpoint endpoint
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"
$ export OSS_ENDPOINT="endpoint"
* remote: fallback to [] if there are no cinfos
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* test: remote: add oss CLI test
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* travis: use iterative's fork of oss-emulator
Just to keep things in-house.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
- Keep current SSH remote connection test on real system
- Faked and Mocked SSH server which running on both Linux and Windows
- Add corresponding tests
Fixed#1704
1) split Project's methods into separate files;
2) rename Project into Repo(similar to gitpython's Repo)
Signed-off-by: Ruslan Kuprieiev <[email protected]>
flake8 provides some additional features like unused vars check, etc, so
it makes total sense to use it as well.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Unfortunately this breaks backward compatibility, but we have to do it,
because current order is totally wrong. Configuring user with `dvc
remote modify myssh user` is an explicit action and should have higher
priority than implicit url. This is also important, since in the next
patch in this PR we will introduce support for `port` configuration,
that should also follow the same logic.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev
Alexander Schepanovski
Jorge OrpinelandRuslan Kuprieiev
pared
nanaya-tachibana
khamutov
Mr. Outis
ag613915cao
ag613915cao
Ivan Shcheklein
Mehdi Hamoumi