Commit Graph
36 Commits
Author SHA1 Message Date
Jorge OrpinelandRuslan Kuprieiev ce83231128 Updates to 1. zsh autocomplete script and 2. slow link type warning message (#2003)
* Document dvc.scm.git::Git::_get_diff_trees and shorten commit hashes in dvc.scm.git::Git::get_diff_trees

* diff: Changes to dvc.scm.git so `dvc diff` outputs short SHAs

- Docstring for Git class constructor
- Renames self.git for self.repo throughout module (involves several test changes)
- Better doc and logical refactor in Git._get_diff_trees
- `dvc diff` now outputs short SHA strings (Git._get_diff_trees) - hard coded to 7 length str
  (involves some test changes in tests/func/test_diff.py)

For #1902

* Undoes "Renames self.git for self.repo throughout module" introduced in 3d2dc83.

* autocomplete: Brings scripts/completion/dvc.zsh up-to-date

- Reorganizes commands in alphabetical order;
- Updates the command dscriptions according to their current usage (-h) output;
- Adds comments for MISSING command entries

* Update scripts/completion/dvc.zsh

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* Update scripts/completion/dvc.zsh

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* Update scripts/completion/dvc.zsh

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* Update scripts/completion/dvc.zsh

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* autocomplete: Adds missing commands to Zsh script

* cache: updates slow link type warning message (and test)

in accordance with https://github.com/iterative/dvc.org/pull/331

* Fixes accidental docstring rewrap

per https://github.com/iterative/dvc/pull/2003#discussion_r286206870

* Actually fixes accidental docstring rewrap...

per https://github.com/iterative/dvc/pull/2003#pullrequestreview-240994592

* Really fixes accidental docstring rewrap this time...
2019-05-25 00:00:47 +03:00
Ruslan Kuprieiev b4143b19dd remote: unify class names
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-12 17:29:38 +03:00
Ruslan Kuprieiev a593a9ce2e dvc: refactor PathInfos
Following same conventions from remotes/outsputs/dependencies.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-12 17:18:46 +03:00
pared 3ff10005fa path_info: remove dict path_info, introduce PathInfo class 2019-05-10 22:25:50 -07:00
nanaya-tachibana 7ceaf88c54 remote: add support for aliyun oss (#1961)
* remote: add support for aliyun oss

Usage:
$ dvc remote add myremote oss://my-bucket.endpoint/path
Set key id and key secret using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"

Ref:
oss python SDK: https://www.alibabacloud.com/help/doc-detail/32026.htm

* Not needed, since we don't support external dependencies/outputs on OSS.
See https://github.com/iterative/dvc/pull/1961.

* Add a way to test oss storage using docker.

Start a container running an oss emulator.
$ git clone https://github.com/nanaya-tachibana/oss-emulator.git
$ docker image build -t oss:1.0 oss-emulator
$ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0
Setup environment variables.
$ export OSS_BUCKET='my-bucket'
$ export OSS_ENDPOINT='localhost:8880'
$ export OSS_ACCESS_KEY_ID='AccessKeyID'
$ export OSS_ACCESS_KEY_SECRET='AccessKeySecret'

* Use default key id and key secret when they are not given,
which gives read access to public read bucket and public bucket.

* test: add oss tests to appveyor.

* remove unneeded tests.

* remote: use s3 style url for oss storage and make endpoint a configurable value.

Usage:
$ dvc remote add myremote oss://my-bucket/path
Set key id, key secret and endpoint using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
$ dvc remote modify myremote oss_endpoint endpoint
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"
$ export OSS_ENDPOINT="endpoint"

* remote: fallback to [] if there are no cinfos

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: remote: add oss CLI test

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* travis: use iterative's fork of oss-emulator

Just to keep things in-house.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-09 15:35:41 +03:00
khamutov 69a1d965e4 test: ssh: update mock version to 3.0.0 min 2019-05-07 11:28:50 +03:00
khamutov 02eebd48eb test: ssh: add windows support for ssh config 2019-05-06 21:47:58 +03:00
khamutov 65fd7f9c8f test: ssh: add support for python 2.7 in mocking 2019-05-06 21:21:22 +03:00
khamutov a4632092a5 test: ssh: add mock_open compat version 2019-05-06 19:21:21 +03:00
khamutov 0afa99028a test: ssh: fixes wrong order of mocks 2019-05-06 09:05:39 +03:00
khamutov 2f1d5d49b4 test: ssh: mock ssh config file exists check 2019-05-05 21:09:05 +03:00
khamutov 0ac256cc96 add tests for ssh config 2019-05-05 19:37:49 +03:00
Ruslan Kuprieiev 5da08289da test: ssh: remove test_connection 2019-05-02 05:56:49 +03:00
Mr. Outis 29f58bda9b status: optimization for already downloaded files
Fix #1839
2019-04-30 17:09:04 -07:00
Ruslan Kuprieiev 435095e35f Merge pull request #1914 from pared/1890
repo: different cache link types info on long lasting operations
2019-04-29 19:14:27 +03:00
pared f496ba4f03 remote: local: different cache link types info on long lasting linking 2019-04-26 10:17:16 -07:00
ag613915cao fd385c913b This commit intends to:
- Keep current SSH remote connection test on real system
- Faked and Mocked SSH server which running on both Linux and Windows
- Add corresponding tests

Fixed #1704
2019-04-26 18:07:16 +07:00
Ruslan Kuprieiev 37ee55b1f8 test: test ssh
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 44b95f23a3 remote: ssh: support directories
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 00cfb04fea remote: generalize file/dir handling
Fixes #1654
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
ag613915cao 47fb2d3a3b Add test for ssh connection
This commit intends to mock SSH server that not depends on
OS which the test running on for testing purpose.
2019-04-21 04:22:59 +03:00
Ivan Shcheklein 75421f0f21 migrate to pytest 2019-03-17 14:49:49 -07:00
Ruslan Kuprieiev 9ca00adde8 dvc: modify flake8 configuration
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-22 19:36:19 +02:00
Ruslan Kuprieiev e5107b1da3 ssh: external-outs: check if cache exists before copying
Fixes #1635

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-19 21:33:47 +02:00
Mr. Outis 5e7a152b53 ssh: use sftp when possible
Related #830 [1/2]
2019-02-18 19:03:06 -06:00
Ruslan Kuprieiev 70c21f7f3e ssh: handle url without path
https://discordapp.com/channels/485586884165107732/485596304961962003/545702950350356502

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-15 04:38:45 +02:00
Ruslan Kuprieiev ba8d83e9b1 dvc: refactor dvc project class
1) split Project's methods into separate files;
2) rename Project into Repo(similar to gitpython's Repo)

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-13 03:31:23 +02:00
Mr. Outis ec77bfa1af remote: define command error on the base
fix: #1484
2019-02-09 12:03:17 -06:00
Mehdi Hamoumi ffc94cc471 Add GCP credentials environment variable to config (#1567) 2019-02-05 01:33:04 +02:00
Ruslan Kuprieiev 4896f08ddf dvc: comply with both flake8 and black
flake8 provides some additional features like unused vars check, etc, so
it makes total sense to use it as well.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-04 00:10:36 +02:00
Ruslan Kuprieiev 4069e34130 dvc: use black instead of flake8
Fixes #1546

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-30 03:48:36 +02:00
Ruslan Kuprieiev a314209add remote: gs: don't forget to set projectname
Fixes #1529

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-26 00:30:08 +02:00
Ruslan Kuprieiev 3053cf2b1f ssh: support specifying port in the url
Fixes #1473

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-08 10:12:43 +02:00
Ruslan Kuprieiev 5d381da5d9 ssh: change order of user configuration
Unfortunately this breaks backward compatibility, but we have to do it,
because current order is totally wrong. Configuring user with `dvc
remote modify myssh user` is an explicit action and should have higher
priority than implicit url. This is also important, since in the next
patch in this PR we will introduce support for `port` configuration,
that should also follow the same logic.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-08 10:12:43 +02:00
Ruslan Kuprieiev 484f318c27 remote: local: support relative paths
Fixes #1421

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-04 21:14:03 +02:00
Ruslan Kuprieiev d3de148fc4 azure: fix backward compatibility
Fixes #1458

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-12-28 19:58:25 +02:00