Jorge Orpinel
1e7efffe42
plot: update some output strings ( #3759 )
...
* plot: update datafile desc.
* plot: link to cmd ref in output
* plot: update --stdout desc
per https://github.com/iterative/dvc/pull/3577#pullrequestreview-407036181
* plot: a few more output string updates
per https://github.com/iterative/dvc/pull/3759#pullrequestreview-409011291
and https://github.com/iterative/dvc.org/pull/1283
2020-05-12 16:52:06 +03:00
Saugat Pachhai
cc16b28a26
diff: autocompletion script for params, and show-md ( #3786 )
2020-05-12 16:41:09 +03:00
Paweł Redzyński
451c7aac04
tests: erepo_dir: extract remote setup to separate method ( #3748 )
2020-05-12 02:05:22 +03:00
Ruslan Kuprieiev
7b4466c982
run-cache: support uncached files ( #3778 )
...
Currently we only restore `--outs`, but `--outs-no-cache` are also
useful, especially when they are metrics files
(`-M|--metrics-no-cache`). So this PR makes run-cache save uncached
files to our cache and also tries to checkout them back during the
`restore` phase.
2020-05-12 02:02:53 +03:00
Jack Wells and Ruslan Kuprieiev
7ca3a1119c
Added S3 SSEKMSKeyId config option ( #3762 )
...
* Added S3 SSEKMSKeyId config option
* test: unit: s3: add SSEKMSKeyId test
Co-authored-by: Ruslan Kuprieiev <[email protected]
2020-05-11 22:52:55 +03:00
Saugat Pachhai
2c026066db
pipeline file: disallow punctuation characters in stage name ( #3767 )
...
except hyphen and underscore
2020-05-11 19:11:47 +03:00
nik123
7f17c6aa03
refactor: check if targets is instances of str ( #3779 )
2020-05-11 19:10:14 +03:00
Ned Letcher
9f015fac1d
import: check erepo is None before modifying url - fixes #3780 ( #3781 )
...
* import: check erepo is None before modifying url - fixes #3780
* import: add test to cover this local import
2020-05-11 18:29:35 +03:00
Saugat Pachhai and Ruslan Kuprieiev
666e203bbb
params/metrics: diff: implement tabulated markdown output (--show-md) ( #3757 )
...
* params/metrics: diff: implement tabulated markdown output
Via `--show-md` on `diff`s.
* table: show None on missing val
* tests: use `dedent`, add tests for markdown
* diff: add --show-md
Co-authored-by: Ruslan Kuprieiev <[email protected]
2020-05-10 18:39:51 +03:00
Karthikeyan Singaravelan
ce946d1160
Import ABC from collections.abc for Python 3 compatibility. ( #3772 )
2020-05-10 10:49:32 +05:45
karajan1001
76b31ac282
fix #3588 ( #3770 )
...
* fix #3588
remove improper edges from output DAG.
* less code modification
* deepsource required
* deepsource required
* refactoring for reducing cognitive complexity.
2020-05-09 21:00:46 +03:00
6093a1fe97
Dvc remote default in list validation ( #3715 )
...
* fix #3470
Add validation to restrict default remote repo in list of remote repos.
* fix #3470
1. add two tests
2. add one validation
3. modified remote remove to satithe validation change
* Pass DeepSource
* deepsource required
* Redesigned.
* fix typo
* Revert "deepsource required"
This reverts commit 1ae458c34f49e8d7b35c39c1e6c592b423493a97.
* Revert "Pass DeepSource"
This reverts commit b254d104df65295e0a1156f3979b365be9d38a3d.
Co-authored-by: karajan1001 <[email protected] [email protected]
2020-05-09 20:59:17 +03:00
Ruslan Kuprieiev
d62a54c7b0
push/pull: properly collect run cache ( #3768 )
...
Fixes #1234
2020-05-08 23:34:35 +03:00
nik123
c922da4f3f
update: --recursive flag ( #3760 )
...
* update: --recursive flag
* refactor: applied isort changes to dvc/repo/update.py
* update: removed default values, check for str
* update: fixed code climate cognitive complexity issues
2020-05-08 23:23:48 +03:00
Ruslan Kuprieiev
2a16b021ad
stage: hide unwanted warnings ( #3763 )
...
These messages all correpond to expected behaviour, so there is no need
to polute the output with unnecessary warnings.
2020-05-08 15:52:09 +03:00
Ruslan Kuprieiev
fc42ca721c
dvc: bump to 1.0.0a1
2020-05-07 19:33:30 +03:00
Ruslan Kuprieiev
785dcfeef0
push/pull: naively transfer run cache ( #3746 )
...
This is the most simple (aka dumb) implementation for this
functionality that is needed to unblock CICD development.
2020-05-07 16:55:10 +03:00
Paweł Redzyński
9714b14456
plot: fix help messages ( #3734 )
2020-05-07 13:22:55 +03:00
Ivan Shcheklein
6a406abde8
trying to enable the sponsorship button
2020-05-06 21:09:45 -07:00
Ivan Shcheklein
7ca055e868
fix: bump PyDrive2 to fix Gdrive remote download
2020-05-06 20:11:33 -07:00
Ruslan Kuprieiev
6a64f86178
pre-commit: use isort seeder
2020-05-07 00:25:41 +03:00
Ruslan Kuprieiev
227dbf44d9
isort: fix config
2020-05-07 00:17:38 +03:00
Casper da Costa-Luis
2bb7257a06
lint: isort pre-commit & misc improvements ( #3749 )
...
* test: clean flake8 config root clutter
* test: pre-commit: add isort
Compatible with black (see https://github.com/timothycrosley/isort/issues/694 )
* lint: reorder imports
* lint: more complex imports
* lint: fix deepsource
* config: isort: add first party dvc
* tests: restyled: add isort
* temp commit to revert before merge
* Revert "temp commit to revert before merge"
This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
Saugat Pachhai
e3831066de
repro: use dvc.yaml by default ( #3747 )
...
Note: This does not change default for `pipeline` as
it cannot show multiple DAGs at the same time. If we
are able to show multiple DAGs, we can just default to
`dvc.yaml`.
2020-05-06 18:33:42 +03:00
Paweł Redzyński
aa99cc9934
plot: check field exists ( #3742 )
2020-05-06 18:33:10 +03:00
Casper da Costa-Luis
26e203242f
gdrive: download: stream & add progress ( #3722 )
...
* wip
* gdrive: add progress
Part of #2865
See https://github.com/iterative/dvc/issues/2865#issuecomment-622984785
* gdrive: move towards next pydrive2 release
- depends on https://github.com/iterative/PyDrive2/pull/30
* update to latest pydrive>=1.4.11
* avoid unneeded API call
* progress: gdrive: ensure proper bar_format
2020-05-06 17:02:16 +03:00
Saugat Pachhai
d00bd4dfe5
pull/fetch/push: add stats ( #3735 )
...
* pull/fetch/push: add stats
* fix typo: s/logging/logger
* pull: make stats detailed, with summary at the bottom
* tests: mock properly, set return_value
* add a line space
2020-05-05 16:10:50 +03:00
Casper da Costa-Luis
795f001929
Snap version ( #3741 )
...
* snap: potential build fix
* snap: fix travis
2020-05-05 12:48:19 +03:00
Ruslan Kuprieiev
93d7b9173b
tests: use --single-stage
2020-05-04 20:23:49 +03:00
Ruslan Kuprieiev
bc7f177c7a
run: use multistage dvcfiles by default ( #3740 )
...
Adding hidden `--single-stage` to simplify the test migration.
2020-05-04 19:55:39 +03:00
Ruslan Kuprieiev
6f73a67c60
dvc: bump to 1.0.0a0
2020-05-04 19:46:49 +03:00
Ruslan Kuprieiev
68a0c39ad1
run: remove deprecated -y|--yes ( #3737 )
2020-05-04 17:51:07 +03:00
Saugat Pachhai
c186a8ea8f
ui: simplify/shorten error messages ( #3730 )
...
* ui: simplify/shorten error messages
* address efiop's review
2020-05-04 17:04:28 +03:00
Paweł Redzyński
af8ac006ad
plot: completion for zsh ( #3733 )
2020-05-04 15:01:29 +03:00
Ruslan Kuprieiev
b2ea279c9e
cache: rename .dvc/cache/run to .dvc/cache/runs ( #3732 )
...
To be consistent with how git does it (e.g. objects) as well as with
future cache structure where we plan on adding `dirs`/`files`/`objects`
subdirs in the future.
2020-05-04 13:26:52 +03:00
Saugat Pachhai
2dda540db6
Make comparisons in is_cached independent of order ( #3731 )
...
On is_cached, we used to compare two dicts: one of current stage in memory
created by `run` and one that's already written to the file.
As `outs` and `dicts` are lists (which is generate by `dumpd`), the
comparison was dependent on the order of outs and dicts.
So, the `run` before and now, can be in different order and would fail
the comparisons.
2020-05-04 11:40:42 +03:00
Ruslan Kuprieiev
76e6ba24f7
run: add to .gitignore on --no-exec ( #3729 )
2020-05-04 02:19:56 +03:00
Ruslan Kuprieiev
505b88afae
diff: make output consistent with other diff commands ( #3727 )
...
* diff: make output consistent with other diff commands
Fixes #3435
* tests: plot: fix flaky test
2020-05-04 00:25:55 +03:00
nik123
415a85c6be
status: --recursive flag ( #3726 )
...
* status: --recursive flag
* tests: fixed test_status_recursive for Windows platform
2020-05-03 21:12:17 +03:00
Ruslan Kuprieiev
4e9b38970a
dvc: move temporary files to .dvc/tmp ( #3725 )
...
Fixes #3666
2020-05-03 18:42:12 +03:00
Ruslan Kuprieiev
3e1eb6d1e7
stage: rename .dvc/cache/stages -> .dvc/cache/run ( #3724 )
2020-05-03 17:19:37 +03:00
Casper da Costa-Luis
c873787d73
progress: add postfix info to avoid overwriting desc ( #3721 )
...
* progress: persist primary description
Move subsequent updates to a postfix.
Clear postfix on exit.
TODO: align nicely.
Fixes #3681 .
* progress: move to posfix[info] for full control
It was bound to happen.
* progress: git: persist description
2020-05-02 18:07:05 +03:00
Paweł Redzyński and Ruslan Kuprieiev
e553511085
Metrics - plotting for multiple revisions initial ( #3577 )
...
* init
* rename to plot data insertion basig on dicts update
* revision support
* roll back revision
* plot makedirs for backward compatibility
* log path
* pretty plot link to visualization page
* make target default title
* efiop review
* efiop review
* plot multiple initial
* add some missing metric file tests
* proper id generation
* proper id generation
* add confusion matrix template
* refactor tests
* plot from dvct file
* plot from dvct
* brush up commands
* fix confusion matrix multiple plot
* plot: change confusion matrix data schema
* should be working as intended
* support for src file in dvct files
* minor fixes
* plot: support json templates
* plot: rename confusion template
* plot: polish command behaviour
* fix test for json
* plot: test command
* some minor fixes for tests
* plot: unit test loading
* plot: unit test loading
* plot: handle TODOS
* cleanup
* use mocker
* plot: support tsv
* plot: command refactoring
* plot: fix windows issues with tests
* plot: test: some more windows fixes
* plot: _load_from_revisions complexity fix
* plot: reduce complexity
* plot: complexity reduction
* plot: deepsource suggestions
* plot: move template path evaluation
* fixup
* fixup
* exception on no datafile and no template
* json metric load with OrderedDict
* plot: improve handling non-existing files on revisions
* plot: improve handling non-existing files on revisions
* change default plot path
* some exceptions and fixes
* add yaml metrics support
* fixup
* some more suggestions
* default filename fix
* efiop review requests
* log exception on failur
* move revisions deduction to commands
* json templates
* extract template filling to separate method
* some parsing improvements
* add columns functionality
* extract default data transformation to separate method
* plot: initial support for jsonpath
* plot: rename columns to filters, tests are dict based
* plot: fixups
* plot: refactoring
* repo: plot: convert to package
* plot: data loading refactor, support searching for data
* plot: raise if wrong fields provided
* plot: command description
* plot: default: pass y axis info for default plot
* plot: get rid of fieldnames, expect ordered data
* plot: handle default plot in separate method
* plot: fix default
* plot: command option names fixes
* refactoring
* fixes
* plot: provide option for stdout redirection
* plot: rename show-json to no-html
* plot: add no-csv-header option
* plot: improve error message for wrongly structured metric
* plot: match template name exactly, whit suffix appended only
* plot: dmpetrov and ivan review
* plot: refactor --stdout help message
* plot: move template to repo/plot
* plot: add -x and -y options
* plot: add -x and -y options
* plot: command: order change
* plot: scatter
* plot: rename confusion matrix template, new name generation format
* plot: add title anchor
* plot: review from jorgeorpinel
* plot: rename filter and result options to select and file
* plot: add --title, --x-title, --y-title
* plot: xlab ylab
* Update dvc/repo/plot/template.py
Co-authored-by: Ruslan Kuprieiev <[email protected] [email protected] [email protected]
2020-05-01 22:45:03 +03:00
Ruslan Kuprieiev and Saugat Pachhai
827c994e74
stage: cache: use lockfiles ( #3713 )
...
* stage: cache: use lockfiles
* reorganize
* Reorganize, use fill_from_lock to load stage cache
* load params from build cache
* adjust tests
Co-authored-by: Saugat Pachhai <[email protected]
2020-05-01 15:23:11 +03:00
Saugat Pachhai and Ruslan Kuprieiev
22c60dd608
dump: deterministic lockfile dump ( #3711 )
...
* dump: lockfile is dumped deterministically
The dump is no longer deterministic/dependent on the pipeline file,
but is sorted based on file names in outs, deps or params.
Also, the params inside each files are also sorted based on name.
However, the objects inside params are not sorted deterministically
as I think it's too much to sort that, and is not easy (considering
the types of objects it might hold, eg: lists, objects, etc).
This will also provide ordered dumps for Python3.5
* fix windows 3.8 test
Co-authored-by: Ruslan Kuprieiev <[email protected]
2020-04-30 22:34:06 +03:00
Ruslan Kuprieiev
f9088e1c67
dvc: rename pipelines.yaml -> dvc.yaml ( #3710 )
2020-04-30 18:19:51 +03:00
Ruslan Kuprieiev
71d156ac49
run: try to save deps before running the command ( #3709 )
...
Unlike old `_check_missing_deps`, this also verifies that we are able to
save more complex dependencies such as parameters, where we not only
care about the config file, but also about the parameters in it.
Fixes #3707
2020-04-30 17:16:16 +03:00
Ruslan Kuprieiev
d998c0a536
serialize: use checksums that are already saved ( #3708 )
...
`get_checksum()` recomputes the checksum which might not match the
pre-recorded one. `checksum` is the one that was `save()`ed during run
and it is the one that should be used in the lockfile.
2020-04-30 16:21:14 +03:00
Peter Rowlands (변기호)
af12f0d06c
remote: adjust traverse threshold multiplier ( #3705 )
...
* Fixes #3704
2020-04-30 12:12:11 +09:00
Ruslan Kuprieiev
18e8f07572
dvc: introduce local build cache ( #3603 )
...
This patch introduces `.dvc/cache/stages` that is used to store previous
runs and their results, which could then be reused later when we stumble
upon the same command with the same deps and outs.
Format of build cache entries is single-line json, which is readable by
humans and might also be used for lock files discussed in #1871 .
Related to #1871
Local part of #1234
2020-04-30 00:22:32 +03:00
Jorge Orpinel
Saugat Pachhai
Paweł Redzyński
Ruslan Kuprieiev
Jack WellsandRuslan Kuprieiev
nik123
Ned Letcher
Karthikeyan Singaravelan
karajan1001
Ivan Shcheklein
Casper da Costa-Luis
Peter Rowlands (변기호)