Ruslan Kuprieiev
ff4fcc937b
stage: fix --dry-run ( #3799 )
...
* tests: unit: move stage tests into a separate dir
* stage: fix --dry-run
Part of #3777
2020-05-14 20:19:34 +05:45
Saugat Pachhai
1676a7a1cd
stage: moving things around, refactor ( #3793 )
...
* stage: extract unnecessary methods out of Stage class
* stage: change naming of functions
* stage: extract dump equal to utils
* stage: extract computation of md5 to utils
* stage: extract command run related logic in run.py
* stage: reorganize run()
* stage: boolean cosmetic changes
* stage: reorder save() and extract some into func
* stage: refactor status() into smaller functions
* extract resolve_wdir into utils
* merge _changed() and changed()
* cosmetic changes
* extract dump to utils
* extract import updates/sync to imports.py
* remove meta from PipelineStage
* fix imports, params load
* make run_stage a function, fix mocks in tests
* use original cmd
* rollback imports on decorators
* fix ds and cc
2020-05-14 16:43:02 +03:00
Ruslan Kuprieiev
974cf097dd
run/repro: rename "build cache" -> "run cache" ( #3788 )
...
Part of #3777
2020-05-14 02:27:18 +03:00
Ruslan Kuprieiev
597ab98d0e
diff: hanle repo with no commits ( #3784 )
...
`git diff` doesn't throw errors, so we could do the same.
Fixes #3398
2020-05-14 02:14:37 +03:00
Peter Rowlands (변기호)
833e82a7ea
Drop Python 3.5 support ( #3792 )
...
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
2020-05-14 01:46:42 +03:00
Peter Rowlands (변기호)
3916d4fa07
tests: mark test_data_cloud::test_pull_git_imports as flaky ( #3790 )
2020-05-14 00:16:37 +09:00
Paweł Redzyński
d81b56978f
plot: add doctype at the beggining of html embedding ( #3787 )
2020-05-13 00:17:05 +03:00
Jorge Orpinel
1e7efffe42
plot: update some output strings ( #3759 )
...
* plot: update datafile desc.
* plot: link to cmd ref in output
* plot: update --stdout desc
per https://github.com/iterative/dvc/pull/3577#pullrequestreview-407036181
* plot: a few more output string updates
per https://github.com/iterative/dvc/pull/3759#pullrequestreview-409011291
and https://github.com/iterative/dvc.org/pull/1283
2020-05-12 16:52:06 +03:00
Saugat Pachhai
cc16b28a26
diff: autocompletion script for params, and show-md ( #3786 )
2020-05-12 16:41:09 +03:00
Paweł Redzyński
451c7aac04
tests: erepo_dir: extract remote setup to separate method ( #3748 )
2020-05-12 02:05:22 +03:00
Ruslan Kuprieiev
7b4466c982
run-cache: support uncached files ( #3778 )
...
Currently we only restore `--outs`, but `--outs-no-cache` are also
useful, especially when they are metrics files
(`-M|--metrics-no-cache`). So this PR makes run-cache save uncached
files to our cache and also tries to checkout them back during the
`restore` phase.
2020-05-12 02:02:53 +03:00
Jack Wells and Ruslan Kuprieiev
7ca3a1119c
Added S3 SSEKMSKeyId config option ( #3762 )
...
* Added S3 SSEKMSKeyId config option
* test: unit: s3: add SSEKMSKeyId test
Co-authored-by: Ruslan Kuprieiev <[email protected]
2020-05-11 22:52:55 +03:00
Saugat Pachhai
2c026066db
pipeline file: disallow punctuation characters in stage name ( #3767 )
...
except hyphen and underscore
2020-05-11 19:11:47 +03:00
nik123
7f17c6aa03
refactor: check if targets is instances of str ( #3779 )
2020-05-11 19:10:14 +03:00
Ned Letcher
9f015fac1d
import: check erepo is None before modifying url - fixes #3780 ( #3781 )
...
* import: check erepo is None before modifying url - fixes #3780
* import: add test to cover this local import
2020-05-11 18:29:35 +03:00
Saugat Pachhai and Ruslan Kuprieiev
666e203bbb
params/metrics: diff: implement tabulated markdown output (--show-md) ( #3757 )
...
* params/metrics: diff: implement tabulated markdown output
Via `--show-md` on `diff`s.
* table: show None on missing val
* tests: use `dedent`, add tests for markdown
* diff: add --show-md
Co-authored-by: Ruslan Kuprieiev <[email protected]
2020-05-10 18:39:51 +03:00
Karthikeyan Singaravelan
ce946d1160
Import ABC from collections.abc for Python 3 compatibility. ( #3772 )
2020-05-10 10:49:32 +05:45
karajan1001
76b31ac282
fix #3588 ( #3770 )
...
* fix #3588
remove improper edges from output DAG.
* less code modification
* deepsource required
* deepsource required
* refactoring for reducing cognitive complexity.
2020-05-09 21:00:46 +03:00
6093a1fe97
Dvc remote default in list validation ( #3715 )
...
* fix #3470
Add validation to restrict default remote repo in list of remote repos.
* fix #3470
1. add two tests
2. add one validation
3. modified remote remove to satithe validation change
* Pass DeepSource
* deepsource required
* Redesigned.
* fix typo
* Revert "deepsource required"
This reverts commit 1ae458c34f49e8d7b35c39c1e6c592b423493a97.
* Revert "Pass DeepSource"
This reverts commit b254d104df65295e0a1156f3979b365be9d38a3d.
Co-authored-by: karajan1001 <[email protected] [email protected]
2020-05-09 20:59:17 +03:00
Ruslan Kuprieiev
d62a54c7b0
push/pull: properly collect run cache ( #3768 )
...
Fixes #1234
2020-05-08 23:34:35 +03:00
nik123
c922da4f3f
update: --recursive flag ( #3760 )
...
* update: --recursive flag
* refactor: applied isort changes to dvc/repo/update.py
* update: removed default values, check for str
* update: fixed code climate cognitive complexity issues
2020-05-08 23:23:48 +03:00
Ruslan Kuprieiev
2a16b021ad
stage: hide unwanted warnings ( #3763 )
...
These messages all correpond to expected behaviour, so there is no need
to polute the output with unnecessary warnings.
2020-05-08 15:52:09 +03:00
Ruslan Kuprieiev
fc42ca721c
dvc: bump to 1.0.0a1
2020-05-07 19:33:30 +03:00
Ruslan Kuprieiev
785dcfeef0
push/pull: naively transfer run cache ( #3746 )
...
This is the most simple (aka dumb) implementation for this
functionality that is needed to unblock CICD development.
2020-05-07 16:55:10 +03:00
Paweł Redzyński
9714b14456
plot: fix help messages ( #3734 )
2020-05-07 13:22:55 +03:00
Ivan Shcheklein
6a406abde8
trying to enable the sponsorship button
2020-05-06 21:09:45 -07:00
Ivan Shcheklein
7ca055e868
fix: bump PyDrive2 to fix Gdrive remote download
2020-05-06 20:11:33 -07:00
Ruslan Kuprieiev
6a64f86178
pre-commit: use isort seeder
2020-05-07 00:25:41 +03:00
Ruslan Kuprieiev
227dbf44d9
isort: fix config
2020-05-07 00:17:38 +03:00
Casper da Costa-Luis
2bb7257a06
lint: isort pre-commit & misc improvements ( #3749 )
...
* test: clean flake8 config root clutter
* test: pre-commit: add isort
Compatible with black (see https://github.com/timothycrosley/isort/issues/694 )
* lint: reorder imports
* lint: more complex imports
* lint: fix deepsource
* config: isort: add first party dvc
* tests: restyled: add isort
* temp commit to revert before merge
* Revert "temp commit to revert before merge"
This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
Saugat Pachhai
e3831066de
repro: use dvc.yaml by default ( #3747 )
...
Note: This does not change default for `pipeline` as
it cannot show multiple DAGs at the same time. If we
are able to show multiple DAGs, we can just default to
`dvc.yaml`.
2020-05-06 18:33:42 +03:00
Paweł Redzyński
aa99cc9934
plot: check field exists ( #3742 )
2020-05-06 18:33:10 +03:00
Casper da Costa-Luis
26e203242f
gdrive: download: stream & add progress ( #3722 )
...
* wip
* gdrive: add progress
Part of #2865
See https://github.com/iterative/dvc/issues/2865#issuecomment-622984785
* gdrive: move towards next pydrive2 release
- depends on https://github.com/iterative/PyDrive2/pull/30
* update to latest pydrive>=1.4.11
* avoid unneeded API call
* progress: gdrive: ensure proper bar_format
2020-05-06 17:02:16 +03:00
Saugat Pachhai
d00bd4dfe5
pull/fetch/push: add stats ( #3735 )
...
* pull/fetch/push: add stats
* fix typo: s/logging/logger
* pull: make stats detailed, with summary at the bottom
* tests: mock properly, set return_value
* add a line space
2020-05-05 16:10:50 +03:00
Casper da Costa-Luis
795f001929
Snap version ( #3741 )
...
* snap: potential build fix
* snap: fix travis
2020-05-05 12:48:19 +03:00
Ruslan Kuprieiev
93d7b9173b
tests: use --single-stage
2020-05-04 20:23:49 +03:00
Ruslan Kuprieiev
bc7f177c7a
run: use multistage dvcfiles by default ( #3740 )
...
Adding hidden `--single-stage` to simplify the test migration.
2020-05-04 19:55:39 +03:00
Ruslan Kuprieiev
6f73a67c60
dvc: bump to 1.0.0a0
2020-05-04 19:46:49 +03:00
Ruslan Kuprieiev
68a0c39ad1
run: remove deprecated -y|--yes ( #3737 )
2020-05-04 17:51:07 +03:00
Saugat Pachhai
c186a8ea8f
ui: simplify/shorten error messages ( #3730 )
...
* ui: simplify/shorten error messages
* address efiop's review
2020-05-04 17:04:28 +03:00
Paweł Redzyński
af8ac006ad
plot: completion for zsh ( #3733 )
2020-05-04 15:01:29 +03:00
Ruslan Kuprieiev
b2ea279c9e
cache: rename .dvc/cache/run to .dvc/cache/runs ( #3732 )
...
To be consistent with how git does it (e.g. objects) as well as with
future cache structure where we plan on adding `dirs`/`files`/`objects`
subdirs in the future.
2020-05-04 13:26:52 +03:00
Saugat Pachhai
2dda540db6
Make comparisons in is_cached independent of order ( #3731 )
...
On is_cached, we used to compare two dicts: one of current stage in memory
created by `run` and one that's already written to the file.
As `outs` and `dicts` are lists (which is generate by `dumpd`), the
comparison was dependent on the order of outs and dicts.
So, the `run` before and now, can be in different order and would fail
the comparisons.
2020-05-04 11:40:42 +03:00
Ruslan Kuprieiev
76e6ba24f7
run: add to .gitignore on --no-exec ( #3729 )
2020-05-04 02:19:56 +03:00
Ruslan Kuprieiev
505b88afae
diff: make output consistent with other diff commands ( #3727 )
...
* diff: make output consistent with other diff commands
Fixes #3435
* tests: plot: fix flaky test
2020-05-04 00:25:55 +03:00
nik123
415a85c6be
status: --recursive flag ( #3726 )
...
* status: --recursive flag
* tests: fixed test_status_recursive for Windows platform
2020-05-03 21:12:17 +03:00
Ruslan Kuprieiev
4e9b38970a
dvc: move temporary files to .dvc/tmp ( #3725 )
...
Fixes #3666
2020-05-03 18:42:12 +03:00
Ruslan Kuprieiev
3e1eb6d1e7
stage: rename .dvc/cache/stages -> .dvc/cache/run ( #3724 )
2020-05-03 17:19:37 +03:00
Casper da Costa-Luis
c873787d73
progress: add postfix info to avoid overwriting desc ( #3721 )
...
* progress: persist primary description
Move subsequent updates to a postfix.
Clear postfix on exit.
TODO: align nicely.
Fixes #3681 .
* progress: move to posfix[info] for full control
It was bound to happen.
* progress: git: persist description
2020-05-02 18:07:05 +03:00
Paweł Redzyński and Ruslan Kuprieiev
e553511085
Metrics - plotting for multiple revisions initial ( #3577 )
...
* init
* rename to plot data insertion basig on dicts update
* revision support
* roll back revision
* plot makedirs for backward compatibility
* log path
* pretty plot link to visualization page
* make target default title
* efiop review
* efiop review
* plot multiple initial
* add some missing metric file tests
* proper id generation
* proper id generation
* add confusion matrix template
* refactor tests
* plot from dvct file
* plot from dvct
* brush up commands
* fix confusion matrix multiple plot
* plot: change confusion matrix data schema
* should be working as intended
* support for src file in dvct files
* minor fixes
* plot: support json templates
* plot: rename confusion template
* plot: polish command behaviour
* fix test for json
* plot: test command
* some minor fixes for tests
* plot: unit test loading
* plot: unit test loading
* plot: handle TODOS
* cleanup
* use mocker
* plot: support tsv
* plot: command refactoring
* plot: fix windows issues with tests
* plot: test: some more windows fixes
* plot: _load_from_revisions complexity fix
* plot: reduce complexity
* plot: complexity reduction
* plot: deepsource suggestions
* plot: move template path evaluation
* fixup
* fixup
* exception on no datafile and no template
* json metric load with OrderedDict
* plot: improve handling non-existing files on revisions
* plot: improve handling non-existing files on revisions
* change default plot path
* some exceptions and fixes
* add yaml metrics support
* fixup
* some more suggestions
* default filename fix
* efiop review requests
* log exception on failur
* move revisions deduction to commands
* json templates
* extract template filling to separate method
* some parsing improvements
* add columns functionality
* extract default data transformation to separate method
* plot: initial support for jsonpath
* plot: rename columns to filters, tests are dict based
* plot: fixups
* plot: refactoring
* repo: plot: convert to package
* plot: data loading refactor, support searching for data
* plot: raise if wrong fields provided
* plot: command description
* plot: default: pass y axis info for default plot
* plot: get rid of fieldnames, expect ordered data
* plot: handle default plot in separate method
* plot: fix default
* plot: command option names fixes
* refactoring
* fixes
* plot: provide option for stdout redirection
* plot: rename show-json to no-html
* plot: add no-csv-header option
* plot: improve error message for wrongly structured metric
* plot: match template name exactly, whit suffix appended only
* plot: dmpetrov and ivan review
* plot: refactor --stdout help message
* plot: move template to repo/plot
* plot: add -x and -y options
* plot: add -x and -y options
* plot: command: order change
* plot: scatter
* plot: rename confusion matrix template, new name generation format
* plot: add title anchor
* plot: review from jorgeorpinel
* plot: rename filter and result options to select and file
* plot: add --title, --x-title, --y-title
* plot: xlab ylab
* Update dvc/repo/plot/template.py
Co-authored-by: Ruslan Kuprieiev <[email protected] [email protected] [email protected]
2020-05-01 22:45:03 +03:00
Ruslan Kuprieiev
Saugat Pachhai
Peter Rowlands (변기호)
Paweł Redzyński
Jorge Orpinel
Jack WellsandRuslan Kuprieiev
nik123
Ned Letcher
Karthikeyan Singaravelan
karajan1001
Ivan Shcheklein
Casper da Costa-Luis