Commit Graph
5269 Commits
Author SHA1 Message Date
Saugat Pachhai 1455e9551e pull/checkout: fix empty output if no state gets changed (#3520) 2020-03-24 13:48:06 +02:00
Saugat Pachhai 9fd7d747c1 pull: support pulling git-only imports (#3503)
* pull: support pulling git-only imports

* pull: also add git imports from external dvc repo

* fetch: add logging

* pull: don't save link to state

When `checkout` or `pull` is done, it'll try to cleanup unused links.
So, when we add from erepo to cache, it creates a link, which DVC will try
to cleanup. Also, it'll provide unwanted checkout change details. So, skipping those

* tests: refactor/rewrite

* flake8: why it's complaining?

* tests: do not leak file descriptors
2020-03-24 13:45:51 +02:00
Alexander SchepanovskiandRuslan Kuprieiev a4094f0a7f dvc: add Repo(skip_checks) flag (#3490)
* dvc: add Repo._skip_graph_checks flag

Will skip modified graph checks on add, run and import. The aim is to
avoid collecting stages and building graph on these commands.

The skip_checks flag is aimed to be private, but let people use dvc on
big repos without monkey patching or writing yaml files themselves.

* dvc: explain skip_graphs_checks flag in a comment

* test: that skip_graph_checks works

* repo: improve hack description and test

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-03-24 01:39:54 +02:00
Ramón Valles ee26afe65e git: quote filenames @ remind_to_track (#3315)
Fix #3303
2020-03-23 18:22:51 +02:00
Jorge Orpinel 9e7740572c dvc: update cmd output strings to match docs (#3471)
* metrics show: update -h output to match docs
per https://github.com/iterative/dvc.org/pull/1014/commits/2c3452119545987f77b764e4aa6fc94d051092a7

* metric: update argument desc per iterative/dvc.org/pull/1014

* metrics: match iterative/dvc.org/pull/1014 again...
using term "formatting" instead of "options"

* list: update help output to match iterative/dvc.org/pull/1014
That's iterative/dvc.org/pull/1014

* dvc: add docstring to open_by_relpath
It's a public fn after all. Similar to Repo class

* help: update cmd output strings to match with docs

* remote: update help output

* term: file descriptor -> file object

* metrics: update add and remove help outputs
per https://github.com/iterative/dvc/pull/3471#pullrequestreview-377170844

* remove: update help output
per https://github.com/iterative/dvc/pull/3471#pullrequestreview-377172394

* root: update help output
per https://github.com/iterative/dvc/pull/3471#pullrequestreview-377173196
2020-03-23 17:40:36 +02:00
Saugat Pachhai 205a605887 external_repo: do not try too hard to fix upstream (#3512)
* tests: importing from bare repo

* external_repo: don't try too hard to set upstream repo especially if url is not a dvc repo.

* Update tests/func/test_import.py

* Add a comment regarding bare-repo clone
2020-03-20 15:54:47 +02:00
Ruslan Kuprieiev 8f546f253b Revert "Restyle #3393 introduce hyper parameters and config (#3516)" (#3517)
This reverts commit 65d22c7dbd.
2020-03-20 00:54:21 +02:00
65d22c7dbd Restyle #3393 introduce hyper parameters and config (#3516)
* Work in progress

* added file parsing and name validation + adjust schema

* Exceptions on bad input

* Support multiple parameters

* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

* Restyled by black

Co-authored-by: elgehelge <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
2020-03-20 00:53:41 +02:00
Ruslan Kuprieiev caf5b931b9 readme: fix flowchart link 2020-03-19 19:21:49 +02:00
Saugat Pachhai 84760dc738 stage: reorganize check for stage paths and exceptions (#3509) 2020-03-19 14:35:12 +02:00
Saugat Pachhai 93246f0658 version: fix bug when .git dir is removed, show more info regarding repo (#3508)
* version: fix error when .git dir is removed

* version: show dvc/git information

* version: remove introducing no_scm params in Repo api
2020-03-19 13:06:34 +02:00
Ivan Shcheklein 4489f98dc3 readme: update flow chart link 2020-03-18 21:06:41 -07:00
Ivan Shcheklein de19d15332 readme: fix banner link after some dvc.org changes 2020-03-18 16:09:49 -07:00
Ruslan Kuprieiev a1fe6c6f44 dvc: bump to 0.90.2 0.90.2 2020-03-18 23:11:17 +02:00
Ruslan Kuprieiev 7edee656da pipeline: fix --outs output (#3507)
Fixes #3497
2020-03-18 23:03:41 +02:00
Ruslan Kuprieiev 498ee509da dvc: use consistent messages for InvalidArgumentError (#3506)
Per https://discordapp.com/channels/485586884165107732/563406153334128681/689896480898810034
2020-03-18 21:34:46 +02:00
Ruslan Kuprieiev 55f86df5b1 dvc: bump to 0.90.1 0.90.1 2020-03-18 19:20:22 +02:00
Ruslan Kuprieiev eae432818a git: tree: use item.path instead of item.name (#3505)
We are currently using this [0] hack to work around the fact that
item.name
is not always a basename. Even though `IndexObject` has it defined that
way
[1], but `Submodule` - doesn't [2]. So using `item.name` might cause
issues
such as [3], because GitPython doesn't pass name parameter when simply
going through objects [4]. In other places, when working specifically
with
submodules, it sets `_name` attribute explicitly [5].

Note that we still don't walk into submodules when walking through a git
repo. Might want to reconsider that later as for now we haven't heard
anything about it from actual users.

Fixes #3481

[0] https://github.com/iterative/dvc/blob/0.90.0/dvc/scm/git/tree.py#L15
[1]
https://github.com/gitpython-developers/GitPython/blob/3.1.0/git/objects/base.py#L170
[2]
https://github.com/gitpython-developers/GitPython/blob/3.1.0/git/objects/submodule/base.py#L1123
[3] https://github.com/gitpython-developers/GitPython/issues/597
[4]
https://github.com/gitpython-developers/GitPython/blob/3.1.0/git/objects/tree.py#L237
[5]
https://github.com/gitpython-developers/GitPython/blob/3.1.0/git/objects/submodule/base.py#L357
2020-03-18 19:18:58 +02:00
Ruslan Kuprieiev aef3dfba13 dvc: bump to 0.90.0 0.90.0 2020-03-18 01:24:19 +02:00
Ruslan Kuprieiev 42c8fab47f version: add supported remotes (#3498)
Since our remotes have optional dependencies, it is really useful to be
able to see what is currently supported. Also helps debug issues with
packages such as brew/conda/etc.
2020-03-18 01:14:59 +02:00
Casper da Costa-Luis f19d800f43 import-url: allow queries in URL (#3432)
* import-url: allow queries in URL

Fixes #3424

* Revert "import-url: allow queries in URL"

This reverts commit 48495d027963073fe260f596d85c3c179fee037f.

* path_info: add HTTPURLInfo base

* path_info: HTTPURLInfo complete

* remote: http(s): support params, queries, fragments

Fixes #3424
Closes #3432

* tests: path_info: HTTPURLInfo

* lint

* tests: path_info: HTTPURLInfo

Slightly more thorough testing
2020-03-17 19:23:56 +02:00
Casper da Costa-LuisandJorge Orpinel 137373c165 template: pr update (#3495)
* template: pr update

* Update .github/PULL_REQUEST_TEMPLATE.md

* Update .github/PULL_REQUEST_TEMPLATE.md

* Update .github/PULL_REQUEST_TEMPLATE.md

Co-authored-by: Jorge Orpinel <[email protected]>
2020-03-17 19:18:25 +02:00
Ruslan Kuprieiev 6559b9d222 dvc: use protected mode by default (#3472)
* dvc: use protected mode by default

Fixes #3261
Fixes #2041

* unprotect: adjust help message

https://github.com/iterative/dvc.org/pull/1058/files/21aab371a487acf6f6e6201b29cd832e7c55ed23#r393324668
2020-03-17 19:16:21 +02:00
Saugat Pachhai 4b5d4d759a gc: do not work without specifier when using -c flag (#3493)
* gc: ask for specifier when using -c flag

* tests: add -w flag with gc -c
2020-03-17 18:12:42 +02:00
Saugat Pachhai e1334448ea completion: add --summary flag on checkout (#3491) 2020-03-17 14:04:12 +02:00
Ruslan Kuprieiev 02daaf391a dvc: bump to 0.89.0 0.89.0 2020-03-17 02:08:56 +02:00
Ruslan Kuprieiev 2ffebb0b5e logger: use standard lazy formatting (#3487)
Unfortunately overriding `format` is not enough in order to use our own
custom formatting in logger. `Record` uses percent formatting that is
hardcoded inside, so we would also have to override it, which is just
way too much effort that will inevitably break as we discover other
methods that are not ready for non-% formatting. So we are just giving
up and sticking to default %-formatting for lazy log messages.

Fixes #3397
2020-03-17 00:38:00 +02:00
Paweł Redzyński 87338c994f cmd: diff: don't load dir cache for diff (#3468) 2020-03-16 21:39:34 +02:00
Saugat PachhaiandCasper da Costa-Luis 109ded4068 dvc: support Python3.8 (#3463)
* py38: make wheels

- ci: set ARROW_LIBHDFS_DIR to load native library
- deps: pin pyarrow to 0.15.1 for py<=3.7
- pin deps for python<3.8
- pin deps for python==3.8 as well
- add windows Py38 on travis
- do not use speedcopy for Python3.8 and Windows

* use realpath to resolve symbolic links

* do not build pkgs for Windows and Linux on Python3.8

* reorder

* pyarrow: document regression fix

Co-authored-by: Casper da Costa-Luis <[email protected]>
2020-03-16 19:18:33 +00:00
Ruslan Kuprieiev 3f3384b7c4 pre-commit: use official dvc repo 2020-03-16 20:41:29 +02:00
Ruslan Kuprieiev b223696079 install: use hooks from master
Same as with shell completion, it is better to use the newest hooks.
2020-03-16 20:40:02 +02:00
c4164b3497 Use pre-commit for Git hooks (#3406)
* Initial design of using pre-commit for Git hooks

* Refactor to reuse existing code and provide merge capabilities

* Linter fixes

* More cleanup

* scm: make use_precommit_tool optional

* dvc: install our own hooks

* install: fix typo

* install: fix bugs in pre-commit integration

* install: don't forget 'always_run' for post-checkout

* install: no need to run pre-commit install if conf file doesn't exist

* install: fix bugs

* install: use python hooks

* add .pre-commit-hooks.yaml

Co-authored-by: Andrew Hare <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-03-16 20:34:01 +02:00
Guro Bokum dcf5af647e list: rename target param to path, update docs (#3462)
Close #3381
2020-03-16 20:22:48 +02:00
Saugat Pachhai 1b21789444 exceptions: use semantic exceptions (#3430) 2020-03-16 19:39:59 +02:00
Saugat PachhaiandJorge Orpinel 066ccea733 checkout: show summary by default and introduce --show-changes flag (#3401)
* checkout: implement --show-changes flag to show detailed checkout details
Show summary by default on checkout

checkout: show directory modified even when files are removed from inside it

checkout: refactor template logic

checkout: change output format

checkout: do not show summary if nothing's changed

checkout: do not show summary if show_changes is specified

checkout: show '/' at the end if it's a directory

rename variable name

test: cover stats for checkout

checkout: show relink as not modified

checkout: remove unwanted message during empty checkout

pull: show summary on checkout

import/get: change external_repo to catch CheckoutError

fix tests and check if path is url

checkout: support external deps/outputs

tests: checkout on external outputs

* fix tests

* checkout: make show changes default and introduce summary flag to show summary

* address suggestions

* exceptions: remove extra spacing on erorr

* Update dvc/command/checkout.py

Co-Authored-By: Jorge Orpinel <[email protected]>

Co-authored-by: Jorge Orpinel <[email protected]>
2020-03-16 16:29:36 +02:00
Saugat Pachhai d2734dfe1e tests: remove use of mocker as context-manager (#3480) 2020-03-16 08:25:22 +05:45
Alexander Schepanovski e69b44d1ed dvc: optimize all target specific commands to not build graph (#3459) 2020-03-15 21:53:38 +02:00
Paweł Redzyński 767433d7a4 tests: add .gitignore files upon dvc_add (#3455) 2020-03-13 17:19:35 +02:00
Saugat Pachhai 682275dc0a add: do not verify hardlink if file is empty (#3428)
* add: do not verify hardlink if file is empty

Fixes #3390

* tests: spy

* fixup! tests: spy
2020-03-11 21:56:12 +02:00
Jorge OrpinelandRuslan Kuprieiev 9944efbd0f api: docstring updates (2) (#3426)
* api: update to latest definitions to match iterative/dvc.org/pull/908

* Update dvc/api.py

* metrics show: update -h output to match docs
per https://github.com/iterative/dvc.org/pull/1014/commits/2c3452119545987f77b764e4aa6fc94d051092a7

* api: update docstrings to match iterative/dvc.org/pull/908

* api: refactor DVC repo check in get_url, and document it

* dvc: cosmetic edits as I explored exceptions that api functions may raise

* api: copy default info to read() docstring from open()
per https://github.com/iterative/dvc/pull/3426#pullrequestreview-366949501

* api: improve open() docstring for clarity and add example
per https://github.com/iterative/dvc/pull/3426#pullrequestreview-366956856

* api: remove unnecessary info from get_url docstring
per https://github.com/iterative/dvc/pull/3426#discussion_r386203064

* api: produces->generated in open() docstring

* api: simplify open docstring
per https://github.com/iterative/dvc/pull/3426#discussion_r388218147

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-03-11 20:45:29 +02:00
Casper da Costa-Luis 7923446219 completion: bash: misc fixes and updates (#3467)
* completion: bash: fix, update and sort

* completion: bash: fix missing `compgen`s

* completion: bash: remove deprecated/hidden options

* completion: bash: fix extra parenthesis

- fixes metrics show -t

* completion: bash: re-add `gc --all-commits`

Due to #3429 superceding #2681
2020-03-11 15:56:20 +02:00
Max Risuhin 3ff804b3b2 gdrive: Google Service account support (#3269) 2020-03-11 15:49:33 +02:00
Ruslan Kuprieiev ef96e6b8ff dvc: bump to 0.88.0 0.88.0 2020-03-10 22:47:01 +02:00
Saugat Pachhai 7f94518008 deps: upgrade jsonpath_ng to latest version (#3464)
Fix #3456
2020-03-10 11:18:52 +02:00
Ruslan Kuprieiev 40519005f5 travis: hadoop: update tar location 2020-03-10 10:20:34 +02:00
Saugat Pachhai ea981ae955 gc: make --all-commits flag visible (#3429)
* gc: make --all-commits visible

* gc: fix help message and completion scripts

* bash: add longform --workspace flag
2020-03-10 08:29:07 +05:45
Ivan Shcheklein 4bceb9d9cd gdrive: share drive vs just folder id when we set corpora param (#3458) 2020-03-09 19:25:08 +02:00
Ruslan Kuprieiev 37e4a5dcd4 travis: use us-west-2 aws s3 region 2020-03-09 19:10:47 +02:00
Ruslan Kuprieiev 6479adb1a3 test: s3: move to dvc-temp bucket 2020-03-09 18:45:59 +02:00
Guro Bokum 02cd520acd api: colorize dvc list (#3351)
Close #3347
2020-03-09 14:07:23 +02:00