Commit Graph
5246 Commits
Author SHA1 Message Date
Saugat Pachhai 4b5d4d759a gc: do not work without specifier when using -c flag (#3493)
* gc: ask for specifier when using -c flag

* tests: add -w flag with gc -c
2020-03-17 18:12:42 +02:00
Saugat Pachhai e1334448ea completion: add --summary flag on checkout (#3491) 2020-03-17 14:04:12 +02:00
Ruslan Kuprieiev 02daaf391a dvc: bump to 0.89.0 0.89.0 2020-03-17 02:08:56 +02:00
Ruslan Kuprieiev 2ffebb0b5e logger: use standard lazy formatting (#3487)
Unfortunately overriding `format` is not enough in order to use our own
custom formatting in logger. `Record` uses percent formatting that is
hardcoded inside, so we would also have to override it, which is just
way too much effort that will inevitably break as we discover other
methods that are not ready for non-% formatting. So we are just giving
up and sticking to default %-formatting for lazy log messages.

Fixes #3397
2020-03-17 00:38:00 +02:00
Paweł Redzyński 87338c994f cmd: diff: don't load dir cache for diff (#3468) 2020-03-16 21:39:34 +02:00
Saugat PachhaiandCasper da Costa-Luis 109ded4068 dvc: support Python3.8 (#3463)
* py38: make wheels

- ci: set ARROW_LIBHDFS_DIR to load native library
- deps: pin pyarrow to 0.15.1 for py<=3.7
- pin deps for python<3.8
- pin deps for python==3.8 as well
- add windows Py38 on travis
- do not use speedcopy for Python3.8 and Windows

* use realpath to resolve symbolic links

* do not build pkgs for Windows and Linux on Python3.8

* reorder

* pyarrow: document regression fix

Co-authored-by: Casper da Costa-Luis <[email protected]>
2020-03-16 19:18:33 +00:00
Ruslan Kuprieiev 3f3384b7c4 pre-commit: use official dvc repo 2020-03-16 20:41:29 +02:00
Ruslan Kuprieiev b223696079 install: use hooks from master
Same as with shell completion, it is better to use the newest hooks.
2020-03-16 20:40:02 +02:00
c4164b3497 Use pre-commit for Git hooks (#3406)
* Initial design of using pre-commit for Git hooks

* Refactor to reuse existing code and provide merge capabilities

* Linter fixes

* More cleanup

* scm: make use_precommit_tool optional

* dvc: install our own hooks

* install: fix typo

* install: fix bugs in pre-commit integration

* install: don't forget 'always_run' for post-checkout

* install: no need to run pre-commit install if conf file doesn't exist

* install: fix bugs

* install: use python hooks

* add .pre-commit-hooks.yaml

Co-authored-by: Andrew Hare <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-03-16 20:34:01 +02:00
Guro Bokum dcf5af647e list: rename target param to path, update docs (#3462)
Close #3381
2020-03-16 20:22:48 +02:00
Saugat Pachhai 1b21789444 exceptions: use semantic exceptions (#3430) 2020-03-16 19:39:59 +02:00
Saugat PachhaiandJorge Orpinel 066ccea733 checkout: show summary by default and introduce --show-changes flag (#3401)
* checkout: implement --show-changes flag to show detailed checkout details
Show summary by default on checkout

checkout: show directory modified even when files are removed from inside it

checkout: refactor template logic

checkout: change output format

checkout: do not show summary if nothing's changed

checkout: do not show summary if show_changes is specified

checkout: show '/' at the end if it's a directory

rename variable name

test: cover stats for checkout

checkout: show relink as not modified

checkout: remove unwanted message during empty checkout

pull: show summary on checkout

import/get: change external_repo to catch CheckoutError

fix tests and check if path is url

checkout: support external deps/outputs

tests: checkout on external outputs

* fix tests

* checkout: make show changes default and introduce summary flag to show summary

* address suggestions

* exceptions: remove extra spacing on erorr

* Update dvc/command/checkout.py

Co-Authored-By: Jorge Orpinel <[email protected]>

Co-authored-by: Jorge Orpinel <[email protected]>
2020-03-16 16:29:36 +02:00
Saugat Pachhai d2734dfe1e tests: remove use of mocker as context-manager (#3480) 2020-03-16 08:25:22 +05:45
Alexander Schepanovski e69b44d1ed dvc: optimize all target specific commands to not build graph (#3459) 2020-03-15 21:53:38 +02:00
Paweł Redzyński 767433d7a4 tests: add .gitignore files upon dvc_add (#3455) 2020-03-13 17:19:35 +02:00
Saugat Pachhai 682275dc0a add: do not verify hardlink if file is empty (#3428)
* add: do not verify hardlink if file is empty

Fixes #3390

* tests: spy

* fixup! tests: spy
2020-03-11 21:56:12 +02:00
Jorge OrpinelandRuslan Kuprieiev 9944efbd0f api: docstring updates (2) (#3426)
* api: update to latest definitions to match iterative/dvc.org/pull/908

* Update dvc/api.py

* metrics show: update -h output to match docs
per https://github.com/iterative/dvc.org/pull/1014/commits/2c3452119545987f77b764e4aa6fc94d051092a7

* api: update docstrings to match iterative/dvc.org/pull/908

* api: refactor DVC repo check in get_url, and document it

* dvc: cosmetic edits as I explored exceptions that api functions may raise

* api: copy default info to read() docstring from open()
per https://github.com/iterative/dvc/pull/3426#pullrequestreview-366949501

* api: improve open() docstring for clarity and add example
per https://github.com/iterative/dvc/pull/3426#pullrequestreview-366956856

* api: remove unnecessary info from get_url docstring
per https://github.com/iterative/dvc/pull/3426#discussion_r386203064

* api: produces->generated in open() docstring

* api: simplify open docstring
per https://github.com/iterative/dvc/pull/3426#discussion_r388218147

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-03-11 20:45:29 +02:00
Casper da Costa-Luis 7923446219 completion: bash: misc fixes and updates (#3467)
* completion: bash: fix, update and sort

* completion: bash: fix missing `compgen`s

* completion: bash: remove deprecated/hidden options

* completion: bash: fix extra parenthesis

- fixes metrics show -t

* completion: bash: re-add `gc --all-commits`

Due to #3429 superceding #2681
2020-03-11 15:56:20 +02:00
Max Risuhin 3ff804b3b2 gdrive: Google Service account support (#3269) 2020-03-11 15:49:33 +02:00
Ruslan Kuprieiev ef96e6b8ff dvc: bump to 0.88.0 0.88.0 2020-03-10 22:47:01 +02:00
Saugat Pachhai 7f94518008 deps: upgrade jsonpath_ng to latest version (#3464)
Fix #3456
2020-03-10 11:18:52 +02:00
Ruslan Kuprieiev 40519005f5 travis: hadoop: update tar location 2020-03-10 10:20:34 +02:00
Saugat Pachhai ea981ae955 gc: make --all-commits flag visible (#3429)
* gc: make --all-commits visible

* gc: fix help message and completion scripts

* bash: add longform --workspace flag
2020-03-10 08:29:07 +05:45
Ivan Shcheklein 4bceb9d9cd gdrive: share drive vs just folder id when we set corpora param (#3458) 2020-03-09 19:25:08 +02:00
Ruslan Kuprieiev 37e4a5dcd4 travis: use us-west-2 aws s3 region 2020-03-09 19:10:47 +02:00
Ruslan Kuprieiev 6479adb1a3 test: s3: move to dvc-temp bucket 2020-03-09 18:45:59 +02:00
Guro Bokum 02cd520acd api: colorize dvc list (#3351)
Close #3347
2020-03-09 14:07:23 +02:00
Alexander Schepanovski 357a6dd467 dvc: optimize building graph (#3443)
Use trie structure to efficiently check out, dep and stage paths being
prefixes of each other.
2020-03-09 12:13:07 +02:00
Ruslan Kuprieiev f05e6e0529 metrics show: update -h output to match docs (#3420)
per https://github.com/iterative/dvc.org/pull/1014/commits/2c3452119545987f77b764e4aa6fc94d051092a7
2020-03-05 16:11:43 +02:00
Ruslan Kuprieiev 06c634adee pipelines: fix edges gathering (#3421) 2020-03-05 14:16:57 +02:00
Alexander Schepanovski 05cc023164 test: check that no remote backend modules are loaded by default (#3400) 2020-02-28 14:45:51 +02:00
Ruslan Kuprieiev abc4e3c2d0 dvc: bump to 0.87.0 0.87.0 2020-02-28 02:21:39 +02:00
Casper da Costa-Luis 35a6e0fedd docs: update badges (#3380)
* docs: add and update badges

Add |Choco|
Add |Brew|
Fix style for |Conda|
Add |PyPI|
Add |Packages|
Fixes #3233
Related https://github.com/iterative/dvc.org/issues/904

* docs: add badge logos

* docs: minor badge tidy

* docs: add choco install, add OS, update order
2020-02-28 01:32:10 +02:00
ttekampeandGuro Bokum de453ab06d dont disable existing loggers when dvc is set up (#3345)
* dont disable existing loggers when dvc is set up

* corrected code style

* dont disable existing loggers, fixes

Co-authored-by: Guro Bokum <[email protected]>
2020-02-27 07:02:06 +02:00
Alexander Schepanovski d8778e7ca9 erepo: set up more logging (#3403) 2020-02-26 22:39:13 +02:00
Bruno Deferrari b136eb5151 erepo: fix relative url on default remote (#2756) (#3378)
* erepo: fix relative urls for remotes (#2756)

* erepo: add test case for relative path remotes (#2756)

* Remove unused argument

* erepo: move handling of relative remotes to a separate method (#2756)

* erepo: don't manually build paths on test (#2756)

* erepo: refactor handling of missing or relative upstream (#2756)

* erepo: move handling of source repo resource to `_fix_upstream` (#2756)

* erepo: improve relative remote tests (#2756)

- use `tmp_dir` fixture
- commit config file changes

* erepo: Remove comment (#2756)

* erepo: use scm_add to commit too, no need for separate scm.commit call (#2756)

* erepo: minor refactoring in `_fix_upstream` (#2756)

* erepo: add comment explaining that old relative paths are restored (#2756)
2020-02-26 22:37:29 +02:00
rxxg d7d5d8fcca Remove Linux pyfastcopy (#3412)
Fixes #3395
2020-02-26 22:35:48 +02:00
Casper da Costa-Luis 68a2f34a46 pull: fetch: external: respect jobs (#3413)
Fixes #3396
2020-02-26 22:35:19 +02:00
Paweł Redzyński a9bc65ee1f analytics: fix resolving used scm (#3405) 2020-02-26 02:29:46 +02:00
Ruslan Kuprieiev 432e399f44 deps: remove pyfastcopy
https://github.com/iterative/dvc/issues/3395
2020-02-25 15:53:43 +02:00
Alexander Schepanovski a81ab5e26b gdrive: do not import pydrive2 on module load (#3399) 2020-02-25 14:52:40 +02:00
Paweł Redzyński 8d04d6d3ce SCM: allow multiple DVC repos inside single SCM repo (#3257) 2020-02-25 14:40:20 +02:00
Jorge OrpinelandRuslan Kuprieiev f673144b19 api: docstring updates (#3352)
* api: update to latest definitions to match iterative/dvc.org/pull/908

* Update dvc/api.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-02-23 06:41:18 +02:00
Ivan Shcheklein eebbf3f812 change diff checksum arg name to hash (#3384) 2020-02-23 06:32:35 +02:00
Alexander Schepanovski 3a5e588561 gdrive: a pair of cleanup refactors (upstream branch) (#3382)
* gdrive: minor clean using funcy 1.14 features

Employ @retry(filter_errors) and @wrap_prop() to pack things up a bit.

* gdrive: refactor dirs/ids/root_id cache and their uses

- cache root transparently
- make cache dirs and ids symmetric, i.e. both using full paths
- implement .list_cache_paths() instead of .all() like in the rest of
the remotes
2020-02-23 06:11:25 +02:00
Saugat Pachhai 5101bf1253 gc: make it safer by only activating by use of certain flags (#3363)
* gc: make it safer by only activating by use of certain flags

Previous plain `gc` is now not supported. The same behavior can be trigger by
`gc -w` or `gc --workspace`. When `--all-{tags, commits, branches}` are
specified, it will work as if `-w` is also specified (same as before).

* tests: adjust gc to use workspace on test_repro
2020-02-21 12:39:59 +02:00
Paweł Redzyński bd33b1ae30 Stage: get checksum from repo if no checksum in stage file (#3365) 2020-02-21 12:10:33 +02:00
68e543f7e1 remotes.http: Support dvc push for http remotes (#3343)
* remote.http: support _upload() via HTTP POST

- uploaded files are sent as chunked encoding POST data

Fixes #3247

* remote.http: add support for HTTP basic and digest auth

- username, password, ask_password options work the same way as for
  ssh remotes
- basic_auth (bool) and digest_auth (bool) options added for http(s)
  remotes
- digest_auth takes precedence over basic_auth if both are enabled

* remote.http: auth_method() unit tests

* tests: Add functional tests for HTTP remotes

- HTTP remotes now tested locally using a SimpleHTTPServer instance that
  allows reading/writing to a temp directory

* tests: fix http tests on py3.5, 3.6

* remote.http: add custom auth method

* s/custom_header/custom_auth_header/

* Apply suggestions from code review

Co-Authored-By: Saugat Pachhai <[email protected]>
Co-Authored-By: Ruslan Kuprieiev <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-02-21 12:05:55 +02:00
Casper da Costa-Luis 411057d1b4 completion: bash: properly recurse targets (#3374)
* completion: bash: properly recurse commands

TODO: subcommands

* completion: bash: add subcommand path completion

* completion: bash: comments
2020-02-21 04:03:23 +02:00
Casper da Costa-Luis ae99e1b02b metadata: add Zenodo, tidy badges and keywords (#3373)
* docs: add Zenodo badge

* docs: fix rst image syntax

* docs: collect and re-use badges

* metadata: add zenodo

Fixes #2978

* metadata: sync keywords

* docs: add citation in readme

* docs: one-line badges
2020-02-21 04:01:59 +02:00