Commit Graph
365 Commits
Author SHA1 Message Date
Jorge OrpinelandRuslan Kuprieiev ce83231128 Updates to 1. zsh autocomplete script and 2. slow link type warning message (#2003)
* Document dvc.scm.git::Git::_get_diff_trees and shorten commit hashes in dvc.scm.git::Git::get_diff_trees

* diff: Changes to dvc.scm.git so `dvc diff` outputs short SHAs

- Docstring for Git class constructor
- Renames self.git for self.repo throughout module (involves several test changes)
- Better doc and logical refactor in Git._get_diff_trees
- `dvc diff` now outputs short SHA strings (Git._get_diff_trees) - hard coded to 7 length str
  (involves some test changes in tests/func/test_diff.py)

For #1902

* Undoes "Renames self.git for self.repo throughout module" introduced in 3d2dc83.

* autocomplete: Brings scripts/completion/dvc.zsh up-to-date

- Reorganizes commands in alphabetical order;
- Updates the command dscriptions according to their current usage (-h) output;
- Adds comments for MISSING command entries

* Update scripts/completion/dvc.zsh

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* Update scripts/completion/dvc.zsh

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* Update scripts/completion/dvc.zsh

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* Update scripts/completion/dvc.zsh

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* autocomplete: Adds missing commands to Zsh script

* cache: updates slow link type warning message (and test)

in accordance with https://github.com/iterative/dvc.org/pull/331

* Fixes accidental docstring rewrap

per https://github.com/iterative/dvc/pull/2003#discussion_r286206870

* Actually fixes accidental docstring rewrap...

per https://github.com/iterative/dvc/pull/2003#pullrequestreview-240994592

* Really fixes accidental docstring rewrap this time...
2019-05-25 00:00:47 +03:00
Ruslan Kuprieiev 3e461f8f11 Merge pull request #2007 from Suor/test-remotes
test: update/add helpers to run tests against remotes
2019-05-21 15:44:23 +03:00
Alexander Schepanovski 0e1f20f0f5 test: update/add helpers to run tests against remotes 2019-05-21 17:38:53 +07:00
Ruslan Kuprieiev 362866ba97 appveyor: specify artifact for specific provider
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-20 16:25:32 +03:00
Ruslan Kuprieiev 7675888a83 appveyor: deploy to pypi
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-20 02:21:35 +03:00
Ruslan Kuprieiev b8fd41e923 travis: deploy to pypi
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-20 01:49:00 +03:00
Ruslan Kuprieiev 797b356e3a pyinstaller: remove hooks that are already released
We've contributed these hooks a long time ago and they got finally
released, so we can remove our copies.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-11 21:44:55 +03:00
Ruslan Kuprieiev aacb0b4c98 pyinstaller: fix build for oss2
This version added support for oss as a remote, which has caused some
issues when building the binary package. To fix those, we need two
pyinstaller hools:

1) hook-Crypto.py
    This one has been added to PyInstaller's upstream, but wasn't
    released yet, so we are using a copy of it for now.

2) hook-aliyunsdkcore.py
    This one is a new one. Sent it to PyInstaller's team in a PR [1].

[1] https://github.com/pyinstaller/pyinstaller/pull/4228

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-11 21:35:41 +03:00
nanaya-tachibana 7ceaf88c54 remote: add support for aliyun oss (#1961)
* remote: add support for aliyun oss

Usage:
$ dvc remote add myremote oss://my-bucket.endpoint/path
Set key id and key secret using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"

Ref:
oss python SDK: https://www.alibabacloud.com/help/doc-detail/32026.htm

* Not needed, since we don't support external dependencies/outputs on OSS.
See https://github.com/iterative/dvc/pull/1961.

* Add a way to test oss storage using docker.

Start a container running an oss emulator.
$ git clone https://github.com/nanaya-tachibana/oss-emulator.git
$ docker image build -t oss:1.0 oss-emulator
$ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0
Setup environment variables.
$ export OSS_BUCKET='my-bucket'
$ export OSS_ENDPOINT='localhost:8880'
$ export OSS_ACCESS_KEY_ID='AccessKeyID'
$ export OSS_ACCESS_KEY_SECRET='AccessKeySecret'

* Use default key id and key secret when they are not given,
which gives read access to public read bucket and public bucket.

* test: add oss tests to appveyor.

* remove unneeded tests.

* remote: use s3 style url for oss storage and make endpoint a configurable value.

Usage:
$ dvc remote add myremote oss://my-bucket/path
Set key id, key secret and endpoint using modify command
$ dvc remote modify myremote oss_key_id my-key-id
$ dvc remote modify myremote oss_key_secret my-key-secret
$ dvc remote modify myremote oss_endpoint endpoint
or environment variables
$ export OSS_ACCESS_KEY_ID="my-key-id"
$ export OSS_ACCESS_KEY_SECRET="my-key-secret"
$ export OSS_ENDPOINT="endpoint"

* remote: fallback to [] if there are no cinfos

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: remote: add oss CLI test

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* travis: use iterative's fork of oss-emulator

Just to keep things in-house.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-09 15:35:41 +03:00
Ruslan Kuprieiev 2cba711237 travis: redirect pyenv log to a file 2019-05-01 02:55:29 +03:00
Mr. Outis c0104e2e61 repro: implement downstream option (#1926)
* repro: implement downstream option

Close #1555

* Update scripts/completion/dvc.zsh

Co-Authored-By: mroutis <[email protected]>

* Update dvc/command/repro.py

Co-Authored-By: mroutis <[email protected]>
2019-04-25 04:50:44 +03:00
Mr. Outis f417f49c6e dvc: deprecate --remove-outs
Close #1853
2019-04-10 18:58:32 -05:00
Ruslan Kuprieiev f97727c785 travis: osx: don't install header package 2019-04-09 16:46:33 +03:00
Ruslan Kuprieiev 458613b797 travis: osx: debug openssl issue 2019-04-09 09:35:00 +03:00
Ruslan Kuprieiev b8e97f5d07 travis: fix xcode version typo 2019-04-08 13:37:53 +03:00
Mr. Outis b4d0d37ac8 dvc: refactor package versioning logic
- Use `version.py` to compute a dynamic version stored in `__version__`.
  Version information was scattered across the source code
  (`VERSION_BASE`, `VERSION`, `__version__`, and `version.py`).

- Read `__version__` from `setup.py` and **pin** it using custom
  `build_py` command class.
  This way, we can maintain consistancy between the build version
  and the runtime one.
  Also, this gives us the ability to run `setup.py` without depending on
  on `dvc` itself.

Close #1824
2019-04-06 18:06:11 -06:00
Ruslan Kuprieiev 2699a47f3e travis: build binary packages only for specific python and os versions
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-04 20:33:39 +03:00
Ruslan Kuprieiev 1c28214728 pyinstaller: hooks: fix google.cloud.storage hook
Fixes #1697

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-03 18:50:15 +03:00
Ruslan Kuprieiev 6ec86e6694 pyinstaller: hook: remove checksumdir hook
That module is not used anymore in dvc.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-03 18:47:28 +03:00
Ruslan Kuprieiev 402eb2da01 travis: debug codecov failure 2019-03-22 15:50:30 +02:00
Mr. Outis ce65771596 scripts: update completion for zsh and bash
Fix #1732
2019-03-21 09:28:54 -07:00
Ruslan Kuprieiev fc646882ad deploy: use xcode 8.3 for mac 2019-03-17 14:38:51 +02:00
Ruslan Kuprieiev 801a492a6b scripts: don't forget sdist for pypi
Fixes #1628

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-18 21:24:49 +02:00
Ruslan Kuprieiev 3c49dfa293 travis: add xcode10.1 build
Related to #1623

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-18 04:03:04 +02:00
Ruslan Kuprieiev c0674dc134 dvc: refactor requirements.txt
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-10 22:42:50 +02:00
Ruslan Kuprieiev 1eee7e3226 travis: enable cron builds
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-09 23:52:15 +02:00
Ruslan Kuprieiev cb8e24237f windows: gen_config: don't use compat open 2019-02-09 07:31:59 +02:00
Ruslan Kuprieiev d3bb4f85df travis: don't forget to init pyenv
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-09 02:12:17 +02:00
Ruslan Kuprieiev 05bd4d4dc4 travis: use pyenv shell
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-08 22:20:41 +02:00
Ruslan Kuprieiev e22da1e957 travis: fix typo
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-08 21:52:13 +02:00
Ruslan Kuprieiev 4f36ce35ad travis: activate pyenv
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-08 21:39:15 +02:00
Mr. Outis 2be7232d1a py2: Add unicode support for open and os.walk 2019-02-07 23:33:30 -06:00
Ruslan Kuprieiev 515211a4dd travis: increase ulimit -u
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-08 02:44:25 +02:00
Ruslan Kuprieiev 92d4acc978 travis: mac: set maxproc to 2048 2019-02-07 06:31:30 +02:00
Ruslan Kuprieiev a95ea2075f travis: debug mac os 2019-02-07 05:27:24 +02:00
Ruslan Kuprieiev 9d663f5035 travis: mac: debug proc limits 2019-02-07 04:40:06 +02:00
Ruslan Kuprieiev 99ef6081c1 test: use creds when testing external gs deps/outs
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-06 00:13:19 +02:00
Ruslan Kuprieiev f6535183a4 travis: gcloud: don't forget to authenticate
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-05 22:50:00 +02:00
Ruslan Kuprieiev 4896f08ddf dvc: comply with both flake8 and black
flake8 provides some additional features like unused vars check, etc, so
it makes total sense to use it as well.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-04 00:10:36 +02:00
Ruslan Kuprieiev 413b938160 travis: osx: upgrade pyenv
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-01 14:55:31 +02:00
Ruslan Kuprieiev 91edbc65f2 dvc: format the rest of the files with black
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-01 14:49:53 +02:00
Ruslan Kuprieiev c5a887dc4e travis: list python versions supported by pyenv
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-01 11:59:23 +02:00
Ruslan Kuprieiev 514b424104 travis: use pyenv to install python on osx
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-01 10:32:58 +02:00
Ruslan Kuprieiev 1aab9e92b4 travis: enable hadoop and azure for non-PR builds
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-31 06:37:36 +02:00
Ruslan Kuprieiev 9fdb922ab7 travis: test azure and hadoop only during cron jobs
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-31 06:03:27 +02:00
Ruslan Kuprieiev 4069e34130 dvc: use black instead of flake8
Fixes #1546

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-30 03:48:36 +02:00
Mr. Outis 0d46921ec7 py2: add unicode literals
Fix #1512
2019-01-27 20:40:21 -06:00
Ruslan Kuprieiev f0cbfe6c72 build: fix deb and rpm packages
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-12-24 18:39:19 +02:00
Ruslan Kuprieiev 0cd680cccf build: don't use system gem 2018-12-24 08:56:23 +02:00
Ruslan Kuprieiev 65d74ee43c build: use --after-{install,remove} scripts
Fixes #1455

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-12-24 08:11:29 +02:00