Commit Graph
5242 Commits
Author SHA1 Message Date
Paweł Redzyński 87338c994f cmd: diff: don't load dir cache for diff (#3468) 2020-03-16 21:39:34 +02:00
Saugat PachhaiandCasper da Costa-Luis 109ded4068 dvc: support Python3.8 (#3463)
* py38: make wheels

- ci: set ARROW_LIBHDFS_DIR to load native library
- deps: pin pyarrow to 0.15.1 for py<=3.7
- pin deps for python<3.8
- pin deps for python==3.8 as well
- add windows Py38 on travis
- do not use speedcopy for Python3.8 and Windows

* use realpath to resolve symbolic links

* do not build pkgs for Windows and Linux on Python3.8

* reorder

* pyarrow: document regression fix

Co-authored-by: Casper da Costa-Luis <[email protected]>
2020-03-16 19:18:33 +00:00
Ruslan Kuprieiev 3f3384b7c4 pre-commit: use official dvc repo 2020-03-16 20:41:29 +02:00
Ruslan Kuprieiev b223696079 install: use hooks from master
Same as with shell completion, it is better to use the newest hooks.
2020-03-16 20:40:02 +02:00
c4164b3497 Use pre-commit for Git hooks (#3406)
* Initial design of using pre-commit for Git hooks

* Refactor to reuse existing code and provide merge capabilities

* Linter fixes

* More cleanup

* scm: make use_precommit_tool optional

* dvc: install our own hooks

* install: fix typo

* install: fix bugs in pre-commit integration

* install: don't forget 'always_run' for post-checkout

* install: no need to run pre-commit install if conf file doesn't exist

* install: fix bugs

* install: use python hooks

* add .pre-commit-hooks.yaml

Co-authored-by: Andrew Hare <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-03-16 20:34:01 +02:00
Guro Bokum dcf5af647e list: rename target param to path, update docs (#3462)
Close #3381
2020-03-16 20:22:48 +02:00
Saugat Pachhai 1b21789444 exceptions: use semantic exceptions (#3430) 2020-03-16 19:39:59 +02:00
Saugat PachhaiandJorge Orpinel 066ccea733 checkout: show summary by default and introduce --show-changes flag (#3401)
* checkout: implement --show-changes flag to show detailed checkout details
Show summary by default on checkout

checkout: show directory modified even when files are removed from inside it

checkout: refactor template logic

checkout: change output format

checkout: do not show summary if nothing's changed

checkout: do not show summary if show_changes is specified

checkout: show '/' at the end if it's a directory

rename variable name

test: cover stats for checkout

checkout: show relink as not modified

checkout: remove unwanted message during empty checkout

pull: show summary on checkout

import/get: change external_repo to catch CheckoutError

fix tests and check if path is url

checkout: support external deps/outputs

tests: checkout on external outputs

* fix tests

* checkout: make show changes default and introduce summary flag to show summary

* address suggestions

* exceptions: remove extra spacing on erorr

* Update dvc/command/checkout.py

Co-Authored-By: Jorge Orpinel <[email protected]>

Co-authored-by: Jorge Orpinel <[email protected]>
2020-03-16 16:29:36 +02:00
Saugat Pachhai d2734dfe1e tests: remove use of mocker as context-manager (#3480) 2020-03-16 08:25:22 +05:45
Alexander Schepanovski e69b44d1ed dvc: optimize all target specific commands to not build graph (#3459) 2020-03-15 21:53:38 +02:00
Paweł Redzyński 767433d7a4 tests: add .gitignore files upon dvc_add (#3455) 2020-03-13 17:19:35 +02:00
Saugat Pachhai 682275dc0a add: do not verify hardlink if file is empty (#3428)
* add: do not verify hardlink if file is empty

Fixes #3390

* tests: spy

* fixup! tests: spy
2020-03-11 21:56:12 +02:00
Jorge OrpinelandRuslan Kuprieiev 9944efbd0f api: docstring updates (2) (#3426)
* api: update to latest definitions to match iterative/dvc.org/pull/908

* Update dvc/api.py

* metrics show: update -h output to match docs
per https://github.com/iterative/dvc.org/pull/1014/commits/2c3452119545987f77b764e4aa6fc94d051092a7

* api: update docstrings to match iterative/dvc.org/pull/908

* api: refactor DVC repo check in get_url, and document it

* dvc: cosmetic edits as I explored exceptions that api functions may raise

* api: copy default info to read() docstring from open()
per https://github.com/iterative/dvc/pull/3426#pullrequestreview-366949501

* api: improve open() docstring for clarity and add example
per https://github.com/iterative/dvc/pull/3426#pullrequestreview-366956856

* api: remove unnecessary info from get_url docstring
per https://github.com/iterative/dvc/pull/3426#discussion_r386203064

* api: produces->generated in open() docstring

* api: simplify open docstring
per https://github.com/iterative/dvc/pull/3426#discussion_r388218147

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-03-11 20:45:29 +02:00
Casper da Costa-Luis 7923446219 completion: bash: misc fixes and updates (#3467)
* completion: bash: fix, update and sort

* completion: bash: fix missing `compgen`s

* completion: bash: remove deprecated/hidden options

* completion: bash: fix extra parenthesis

- fixes metrics show -t

* completion: bash: re-add `gc --all-commits`

Due to #3429 superceding #2681
2020-03-11 15:56:20 +02:00
Max Risuhin 3ff804b3b2 gdrive: Google Service account support (#3269) 2020-03-11 15:49:33 +02:00
Ruslan Kuprieiev ef96e6b8ff dvc: bump to 0.88.0 0.88.0 2020-03-10 22:47:01 +02:00
Saugat Pachhai 7f94518008 deps: upgrade jsonpath_ng to latest version (#3464)
Fix #3456
2020-03-10 11:18:52 +02:00
Ruslan Kuprieiev 40519005f5 travis: hadoop: update tar location 2020-03-10 10:20:34 +02:00
Saugat Pachhai ea981ae955 gc: make --all-commits flag visible (#3429)
* gc: make --all-commits visible

* gc: fix help message and completion scripts

* bash: add longform --workspace flag
2020-03-10 08:29:07 +05:45
Ivan Shcheklein 4bceb9d9cd gdrive: share drive vs just folder id when we set corpora param (#3458) 2020-03-09 19:25:08 +02:00
Ruslan Kuprieiev 37e4a5dcd4 travis: use us-west-2 aws s3 region 2020-03-09 19:10:47 +02:00
Ruslan Kuprieiev 6479adb1a3 test: s3: move to dvc-temp bucket 2020-03-09 18:45:59 +02:00
Guro Bokum 02cd520acd api: colorize dvc list (#3351)
Close #3347
2020-03-09 14:07:23 +02:00
Alexander Schepanovski 357a6dd467 dvc: optimize building graph (#3443)
Use trie structure to efficiently check out, dep and stage paths being
prefixes of each other.
2020-03-09 12:13:07 +02:00
Ruslan Kuprieiev f05e6e0529 metrics show: update -h output to match docs (#3420)
per https://github.com/iterative/dvc.org/pull/1014/commits/2c3452119545987f77b764e4aa6fc94d051092a7
2020-03-05 16:11:43 +02:00
Ruslan Kuprieiev 06c634adee pipelines: fix edges gathering (#3421) 2020-03-05 14:16:57 +02:00
Alexander Schepanovski 05cc023164 test: check that no remote backend modules are loaded by default (#3400) 2020-02-28 14:45:51 +02:00
Ruslan Kuprieiev abc4e3c2d0 dvc: bump to 0.87.0 0.87.0 2020-02-28 02:21:39 +02:00
Casper da Costa-Luis 35a6e0fedd docs: update badges (#3380)
* docs: add and update badges

Add |Choco|
Add |Brew|
Fix style for |Conda|
Add |PyPI|
Add |Packages|
Fixes #3233
Related https://github.com/iterative/dvc.org/issues/904

* docs: add badge logos

* docs: minor badge tidy

* docs: add choco install, add OS, update order
2020-02-28 01:32:10 +02:00
ttekampeandGuro Bokum de453ab06d dont disable existing loggers when dvc is set up (#3345)
* dont disable existing loggers when dvc is set up

* corrected code style

* dont disable existing loggers, fixes

Co-authored-by: Guro Bokum <[email protected]>
2020-02-27 07:02:06 +02:00
Alexander Schepanovski d8778e7ca9 erepo: set up more logging (#3403) 2020-02-26 22:39:13 +02:00
Bruno Deferrari b136eb5151 erepo: fix relative url on default remote (#2756) (#3378)
* erepo: fix relative urls for remotes (#2756)

* erepo: add test case for relative path remotes (#2756)

* Remove unused argument

* erepo: move handling of relative remotes to a separate method (#2756)

* erepo: don't manually build paths on test (#2756)

* erepo: refactor handling of missing or relative upstream (#2756)

* erepo: move handling of source repo resource to `_fix_upstream` (#2756)

* erepo: improve relative remote tests (#2756)

- use `tmp_dir` fixture
- commit config file changes

* erepo: Remove comment (#2756)

* erepo: use scm_add to commit too, no need for separate scm.commit call (#2756)

* erepo: minor refactoring in `_fix_upstream` (#2756)

* erepo: add comment explaining that old relative paths are restored (#2756)
2020-02-26 22:37:29 +02:00
rxxg d7d5d8fcca Remove Linux pyfastcopy (#3412)
Fixes #3395
2020-02-26 22:35:48 +02:00
Casper da Costa-Luis 68a2f34a46 pull: fetch: external: respect jobs (#3413)
Fixes #3396
2020-02-26 22:35:19 +02:00
Paweł Redzyński a9bc65ee1f analytics: fix resolving used scm (#3405) 2020-02-26 02:29:46 +02:00
Ruslan Kuprieiev 432e399f44 deps: remove pyfastcopy
https://github.com/iterative/dvc/issues/3395
2020-02-25 15:53:43 +02:00
Alexander Schepanovski a81ab5e26b gdrive: do not import pydrive2 on module load (#3399) 2020-02-25 14:52:40 +02:00
Paweł Redzyński 8d04d6d3ce SCM: allow multiple DVC repos inside single SCM repo (#3257) 2020-02-25 14:40:20 +02:00
Jorge OrpinelandRuslan Kuprieiev f673144b19 api: docstring updates (#3352)
* api: update to latest definitions to match iterative/dvc.org/pull/908

* Update dvc/api.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-02-23 06:41:18 +02:00
Ivan Shcheklein eebbf3f812 change diff checksum arg name to hash (#3384) 2020-02-23 06:32:35 +02:00
Alexander Schepanovski 3a5e588561 gdrive: a pair of cleanup refactors (upstream branch) (#3382)
* gdrive: minor clean using funcy 1.14 features

Employ @retry(filter_errors) and @wrap_prop() to pack things up a bit.

* gdrive: refactor dirs/ids/root_id cache and their uses

- cache root transparently
- make cache dirs and ids symmetric, i.e. both using full paths
- implement .list_cache_paths() instead of .all() like in the rest of
the remotes
2020-02-23 06:11:25 +02:00
Saugat Pachhai 5101bf1253 gc: make it safer by only activating by use of certain flags (#3363)
* gc: make it safer by only activating by use of certain flags

Previous plain `gc` is now not supported. The same behavior can be trigger by
`gc -w` or `gc --workspace`. When `--all-{tags, commits, branches}` are
specified, it will work as if `-w` is also specified (same as before).

* tests: adjust gc to use workspace on test_repro
2020-02-21 12:39:59 +02:00
Paweł Redzyński bd33b1ae30 Stage: get checksum from repo if no checksum in stage file (#3365) 2020-02-21 12:10:33 +02:00
68e543f7e1 remotes.http: Support dvc push for http remotes (#3343)
* remote.http: support _upload() via HTTP POST

- uploaded files are sent as chunked encoding POST data

Fixes #3247

* remote.http: add support for HTTP basic and digest auth

- username, password, ask_password options work the same way as for
  ssh remotes
- basic_auth (bool) and digest_auth (bool) options added for http(s)
  remotes
- digest_auth takes precedence over basic_auth if both are enabled

* remote.http: auth_method() unit tests

* tests: Add functional tests for HTTP remotes

- HTTP remotes now tested locally using a SimpleHTTPServer instance that
  allows reading/writing to a temp directory

* tests: fix http tests on py3.5, 3.6

* remote.http: add custom auth method

* s/custom_header/custom_auth_header/

* Apply suggestions from code review

Co-Authored-By: Saugat Pachhai <[email protected]>
Co-Authored-By: Ruslan Kuprieiev <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-02-21 12:05:55 +02:00
Casper da Costa-Luis 411057d1b4 completion: bash: properly recurse targets (#3374)
* completion: bash: properly recurse commands

TODO: subcommands

* completion: bash: add subcommand path completion

* completion: bash: comments
2020-02-21 04:03:23 +02:00
Casper da Costa-Luis ae99e1b02b metadata: add Zenodo, tidy badges and keywords (#3373)
* docs: add Zenodo badge

* docs: fix rst image syntax

* docs: collect and re-use badges

* metadata: add zenodo

Fixes #2978

* metadata: sync keywords

* docs: add citation in readme

* docs: one-line badges
2020-02-21 04:01:59 +02:00
Ruslan Kuprieiev 2c46bddcfa dvc: bump to 0.86.5 0.86.5 2020-02-21 00:11:41 +02:00
Ruslan Kuprieiev e4bf5c1ddb dvc: bump to 0.86.4 0.86.4 2020-02-20 03:46:51 +02:00
Ruslan Kuprieiev 6474725d4f test: repro: prevent dvc.stages resetting (#3368) 2020-02-20 03:45:28 +02:00
Ruslan Kuprieiev c5677decd3 dvc: bump to 0.86.3 0.86.3 2020-02-20 02:40:14 +02:00