Commit Graph
5576 Commits
Author SHA1 Message Date
Casper da Costa-Luis fa0f182028 completion: autogenerate (#3921)
* cli: split main_parser

* cli: basic parser tree printing

* cli: major completion tidy

* completion: more missing root commands

* completion: much automagic such beauty

- fixes #2985 automatically generate bash completion

* tests: bash completion

* tests: completion: tidy script a little

* minor tidy

* command: positional name consistency

* completion: bash: add logging

* double-check renames and consistency

* cli: add completion subcommand

* ci: completion: update test

* minor logging

* more completion logging verbosity

* fix completion of files

* add shtab

* purge unneeded completion => shtab

* completion: neaten UI

* completion: use `shtab>=0.0.2`

* completion: purge hardcoded completions!

* revert unneeded renames for now

* fix isort

* completion: tidy CLI API

* fix snap build

* fix snap build again

* snap: internally build completion

* allow repo-less completion

* command: tidy headless logic

* use CmdBaseNoRepo

* better DVC-file completion

* merge command.choices -> command.completion

* drop choices class

* remove completion optionals

* tidy docs
2020-06-19 04:54:20 +03:00
Ruslan Kuprieiev 15c787f2ac innosetup: commit images 2020-06-18 23:39:33 +03:00
Saugat Pachhai 4af3919062 Add flake8 extensions (#4067)
* remove unnecessary generators/comprehensions

* Use dict literals

* remove unused variables

* misc fixes

ignore .iteritems, is a false +ve
remove function call in default argument

* Flake8 config: complexity, show source on err, show err counts

* flake8: add extensions

* remove flake8-string-format

* Add flake8-debugger support

* Add flake8-string-format extension

* remove flake8-debugger

It's having some problems to install on Windows in Travis
2020-06-18 14:36:30 +03:00
Peter Rowlands (변기호) 3a469f2e93 dvcignore: ignore empty (None) patterns (#4066) 2020-06-18 14:31:30 +09:00
Ruslan Kuprieiev 51a8c782ae tests: create remote fixtures (#4063)
* tests: use remote fixtures

* tests: use remote fixtures for data cloud tests

* tests: treat remotes as tmp dirs
2020-06-18 01:36:13 +03:00
Ruslan Kuprieiev b579d06c8e dvc: bump to 1.0.0b5 1.0.0b5 2020-06-18 00:33:53 +03:00
Alexander Schepanovski 039cd49259 plots: clean up template anchors and collection (#4061) 2020-06-17 21:34:43 +03:00
Peter Rowlands (변기호) f516dd252a trees: handle tree.walk errors the same way as os.walk (#4059)
* trees: BaseTree.walk should handle errors like os.walk

* use onerror callback if exceptions should be logged or raised

* repo.ls: use onerror for tree.walk
2020-06-17 13:25:14 +03:00
Banti Gheneti 706dd05e8d remote/gs: fix url signing when using GoogleStorage on GoogleComputeEngine (#4057)
Fixes #4056
2020-06-17 13:24:02 +03:00
Saugat Pachhai 5d957b1830 cache relpath and path_in_repo (#4053) 2020-06-17 04:28:10 +03:00
Saugat Pachhai 387867fbbe optimize: remove fsync when dumping rwlock data (#4052)
Fixes #3653
2020-06-16 17:55:36 +03:00
Ruslan Kuprieiev 7c6351d039 dvc: bump to 1.0.0b4 1.0.0b4 2020-06-16 15:17:19 +03:00
Ruslan Kuprieiev 613ca0bdb5 get-url/import-url: save dependency before downloading (#4049)
This allows us to properly process the errors like missing target.

Fixes #3897
2020-06-16 13:05:21 +03:00
Peter Rowlands (변기호) 0fb9870de6 remote: rename "checksum" -> "hash" (#4051)
* remote: s/get_checksum/get_hash/

* remote/tree: s/get_file_checksum/get_file_hash/

* remote: s/get_dir_checksum/get_dir_hash/

* remote: s/checksum_to_path/hash_to_path/

* remote: s/path_to_checksum/path_to_hash/

* remote: cleanup remaining checksum references

* remote.index: s/checksum/hash/

* note: does not rename "checksum" field in the index DB schema
2020-06-16 12:16:51 +03:00
Ruslan Kuprieiev 73e60a8c13 Delete pre-commit.yaml 2020-06-15 23:35:10 +03:00
Ruslan Kuprieiev 9d1dde4051 Create pre-commit.yaml 2020-06-15 23:32:39 +03:00
Ruslan Kuprieiev 38dc0f85d4 dvc: bump to 1.0.0b3 1.0.0b3 2020-06-15 22:15:57 +03:00
Paweł Redzyński d2dbe9649c plots: add smooth template (#4046) 2020-06-15 22:15:27 +03:00
Ruslan Kuprieiev ece6f2d3ee dvc: bump to 1.0.0b2 1.0.0b2 2020-06-15 21:36:05 +03:00
Ruslan Kuprieiev b672622324 plots: make sure template is using data anchor (#4045)
Fixes #3969
2020-06-15 21:14:07 +03:00
Ruslan Kuprieiev c9391df313 plots: modify: handle --unset for nonexistent options (#4044)
* tests: plots: reorginize the structure

* plots: modify: handle --unset for nonexistent options

Fixes #3933
2020-06-15 20:31:08 +03:00
Peter Rowlands (변기호) a0c229cf9b remote: continue remote/cache/tree refactoring (#4041)
* remote: move constants into trees

* remote: replace Remote() Cache() helper functions

* output: update for remote changes

* cache/data_cloud: update for remote changes

* dependency: update for remote changes

* stage: update for remote changes

* remote: move remaining non-index methods into tree

* tests: update remote unit tests

* remote: LocalCache needs to lock remote index, not cache index

* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH

* tests: repo tree open needs state context

* remote: make Remote.gc a class method

* operates on obj.tree, so that GC can be done with both remote and
  cache instances

* bug fixes

* tests: update test_data_cloud

* bug fixes

* tests: update func tests
2020-06-15 16:22:55 +03:00
karajan1001 a59f90fac9 Performance improvement of dvcignore (#3967)
fix#3869
1.Use big regex.

* Solve windows

* add rule order test

* Solve ignore order

* remove list comprehensions
2020-06-15 11:37:01 +02:00
Ruslan Kuprieiev bba502314d dvc: bump to 1.0.0b1 1.0.0b1 2020-06-14 20:44:44 +03:00
Saugat Pachhai bc4bf66d17 (bugfix) get/import: fix broken progress bar (#4039)
PR #3672 (6d8499e) extended `LocalRemote::_get_plans` to return one
`checksums` too. As all of the args from `_get_plans` was passed
down to `download()`, it recognized extra arg of checksum as
`no_progress_bar` due to which it became True and stopped showing
progress bar at all.

Fix #3874
2020-06-14 20:42:53 +03:00
f7348af1e0 metrics & plots: help output improvements (#3963)
* metrics & plots: help output improvements

* metrics & plots: ui: updated help messages

* refactor: removed redundant parentheses

* metrics: ui: updated 'targets' description in 'metrics show'

* metrics: ui: reordered params to match the order in cmd ref

* plots: ui: params modified to match cmd ref

* plots: ui: updated diff help

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
2020-06-14 20:39:33 +03:00
Casper da Costa-Luis 4d140a9944 gdrive: add open (#3916)
* gdrive: add open

Fixes #3408
Related #2865
Fixes #3897

* dependency: add gdrive

* test: api: open: gdrive

* rollback pydrive dependency

* Revert "dependency: add gdrive"

This reverts commit fd333261d2e9fe1364cc278138c79890a1f5fe7a.

* tests: remotes: GDrive: fix get_url

* tests: api: fully test GDrive

* minor typo fix

* attempt gdrive tests fix

* fix gdrive credentials config

* replace GDrivePathNotFound => FileMissingError

* fix test_open_external[GDrive]

* add ensure_dir_scm

* minor exception fix
2020-06-13 23:53:25 +03:00
Casper da Costa-Luis 2ba73daedc snap: make future major version upgrades easier (#4038)
Related: #3999
Related: #4037
2020-06-13 23:27:03 +03:00
Ruslan Kuprieiev 7a31a35cbf dvc: bump to 1.0.0b0 1.0.0b0 2020-06-13 00:56:14 +03:00
Saugat Pachhai 24a14563cc run: preserve order when -f/--force overwrite stage (#4035)
This commit preserves comments, meta and order inside a dvc.yaml
when `-f/--force` is used. But, it does not maintain comments/meta
for .dvc files.

Related: #4014
2020-06-13 00:55:15 +03:00
Ruslan Kuprieiev a1d3440f85 metrics: diff: introduce --precision (#4026)
Rounding to 5 digits by default.

Fixes #3904
2020-06-13 00:54:42 +03:00
Saugat Pachhai 45dcb7c27d Update dag.py (#4036) 2020-06-12 20:33:19 +03:00
Saugat Pachhai 9ff6fef53a bugfix: get dump folder contents in respective output folder (#4034)
The relative path for external git directory was computed relative to
the root of the repo itself, which created a relative path which is
current working directory. So, all of the contents of the erepo to download
was dumped in the pwd rather than inside of it's own directory.

Fix #4033
2020-06-12 20:31:18 +03:00
Saugat PachhaiandRuslan Kuprieiev 9e7dc2640e schema: provide a jsonschema for dvc.yaml (#4028)
* schema: provide a jsonschema for dvc.yaml

* Update scripts/schema/dvc-yaml.json

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update scripts/schema/dvc-yaml.json

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-06-12 20:09:59 +05:45
Paweł Redzyński 584c97a58d plots: fix handling of multiple plots from single revision (#4031) 2020-06-12 16:08:08 +03:00
Saugat Pachhai b94a18cfb9 tests: tests for dvcfile (#4032) 2020-06-12 15:54:05 +03:00
Peter Rowlands (변기호) 9ead641c95 remote: separate cloud remote and cloud cache classes (#4019)
* remote: move get_file_checksum() into tree

* makes RemoteTree and RepoTree consistent with regard to checksum
  calculation

* remote: save() now takes explicit tree parameter

* when tree is remote.tree, save will be a move + link operation
  (same as default existing behavior)
* when saving path from a different tree, save will be a copy operation

* tests: update for moved remote/tree functions

* remote: move get_checksum into tree

* remote: separate cloud remote and cache classes

* tests: update unit tests for remote/cache separation

* remote: cloud cache should extend cloud remote

* remote: move LocalRemote.get

* tests: update func tests

* dependency: update for moved get_checksum

* remote: fix state lookup bug
2020-06-12 15:52:47 +03:00
Peter Rowlands (변기호) 652f5ab439 remote: use separate working tree instance for local cache (#3991)
* dvcignore: ignore paths outside of CleanTree root

* remote: local remote/cache should always use its own working tree

* repo tree (CleanTree) instance should not be relevant to local cache
  operations. When the dvcignore CleanTree is relevant, repo.tree should
  be used. For local remote/cache operations, the local remote work_tree
  should be used.

* state: tie state to local cache working tree

* tests: update tests

* state: use os.path.exists instead of WorkingTree.exists in some cases
2020-06-12 12:52:38 +03:00
bjonnhandJonathan Bisson 28a123e727 Remove a remnant umask to solve #4021 (#4022)
Co-authored-by: Jonathan Bisson <[email protected]>
2020-06-11 19:47:25 +03:00
Ruslan Kuprieiev 566472f088 dvc: bump to 1.0.0a11 1.0.0a11 2020-06-11 19:09:51 +03:00
Saugat PachhaiandRuslan Kuprieiev 2e12a31c5a tests: cover run cache with functional tests (#4018)
* tests: cover run cache with functional tests

* run-cache: use _log_exceptions wrapper

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-06-11 18:44:21 +03:00
Alexander Schepanovski d7788887f0 plots: fix a couple of bugs (#4017)
- running .plots.collect() not from repo root used to skip plot files
- make Plots.render() be more convenient to be called from outside
2020-06-11 18:20:12 +07:00
Ruslan Kuprieiev 87276faef2 RepoTree: work with dirty repos (#4005)
Currently `RepoTree.open` will throw FileNotFoundError if you have a
file that is tracked by dvc, but which has no md5 or a cache for which
is missing, even though you have the file itself in your workspace.

This corresponds very nicely to the `--no-commit` or `--no-exec`
scenario, where you might not want to commit your artifacts to cache
just yet.

Fixes #3974
2020-06-11 09:50:27 +03:00
Saugat Pachhai 4042d5232d run: use -f/--force to overwrite, fix msg (#4011)
Also, makes fixes for `--overwrite` rename to `--force`
and also remove `remove_with_prompt` which was only being
used with `force\=True` anyway.

Fix #4003
2020-06-10 20:17:27 +03:00
Saugat Pachhai fda14c82d3 plots modify: do not convert output types to plot on unset (#4009)
Fix #4006
2020-06-10 15:31:17 +03:00
Saugat Pachhai 64be262d18 dvc.yaml: preserve meta (#4008) 2020-06-10 14:37:21 +03:00
Saugat Pachhai 1d6c80001d plots modify: complain if template does not exist (#3995) 2020-06-10 16:12:56 +05:45
Ruslan Kuprieiev 91a7194e4f add/run/import-url: don't use -f as --file (#4004)
If anything, `-f` is closer to `--force` than to `--file`. So let's set
this straight.
2020-06-10 05:04:55 +03:00
Ruslan Kuprieiev afe539aaba plots: rename --no-csv-header to --no-header (#4001)
* plots: rename --no-csv-header to --no-header

Fixes #3996

* fix typo
2020-06-10 03:34:25 +03:00
Ruslan Kuprieiev f259bc52cd config/remote/cache: make --system/--global/--local mutually exclusive (#4000) 2020-06-10 02:32:18 +03:00