Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,777 Commits 7 Branches 579 Tags
9ff18502b612656de47c63aade618d3a5be11b70
Commit Graph
82 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
David de la Iglesia Castro f95e6340e0 utils: remove dead code 2022-07-22 16:30:51 +02:00
David de la Iglesia Castro ed1d8e2b8f exp: init: Use persist: true if checkpoint. 2022-07-21 13:21:06 +02:00
David de la Iglesia Castro c2668110b7 gc: Add --rev and --num. 
Closes #7695
 2022-07-12 20:13:07 +02:00
David de la Iglesia Castro d47e3c40b8 tests: limit printed durations to 50 2022-07-11 12:36:37 +03:00
David de la Iglesia Castro 954de0cae4 api: params_show: Raise exception if no params found. 
Fixes #7926
 2022-07-05 09:18:22 +02:00
David de la Iglesia Castro 909941308b parsing: Support dict unpacking in cmd. 
Allow to use dictionaries as values for template interpolation but only inside the `cmd` key.

See tests/func/parsing/test_interpolated_entry.py::test_cmd_dict for detailed syntax.

Add `config.parsing` section for configuring behavior of ambiguous data types
like booleans and lists.
 2022-07-05 09:02:15 +02:00
David de la Iglesia Castro de8ab044f8 Downgrade pyinstaller 
Per https://github.com/iterative/dvc/issues/7949#issuecomment-1170927010
 2022-06-30 15:24:11 +03:00
David de la Iglesia Castro c336507e51 render: image_converter: Support slash in revision. 
Fixes #7934
 2022-06-27 10:53:03 +02:00
David de la Iglesia Castro 20f69a121e api: Move to sub-package. 
Split into sub-modules.

Split tests.
 2022-06-21 20:52:19 +03:00
David de la Iglesia Castro b994c529a2 api: Add params_show. 
Closes #6507

Uses `repo.params.show` with custom error_handler and postprocess the outputs for more user-friendly structure.

Extend `repo.params.show` to accept `stages` argument to cover the "params of current stage" use case.
 2022-06-21 20:52:19 +03:00
David de la Iglesia Castro f106a68424 params.show: Fix deps for stages using same params file. 2022-06-21 20:52:19 +03:00
David de la Iglesia Castro 66c2e8cbd8 repo.collect: Add duplicates option. 
Defaults to `False`.
If `True`, multiple `outs` sharing a provided `target_path` will not be filtered.
 2022-06-21 20:52:19 +03:00
David de la Iglesia CastroandPeter Rowlands 1663b97eaa api: open: Raise ValueError if rev is used in wrong mode. 
Closes #7405

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
 2022-06-15 18:19:27 +02:00
David de la Iglesia Castro 15e10a1d81 api: open: Add docstring and type hints. 2022-06-15 18:19:27 +02:00
David de la Iglesia Castro 3d791778de plots: Pass templates_dir to match_renderers. 
We were forgetting to pass default template dir.

Fixes #7817.
 2022-05-31 16:17:58 +02:00
David de la Iglesia Castro 34ef1807be exp: show: Include additional info in --json. 
Use relpath to repo.root_dir as keys in deps and outs.
Add `use_cache` and `is_data_source` for outs.

This allows to differentiate git-tracked dependencies and
intermediate outputs.

Closes #7575
Closes #7790
 2022-05-25 09:01:39 +02:00
David de la Iglesia Castro 109d0e4e28 dag: mermaid: Use quotation marks to escape node name. 
Fixes #7731
 2022-05-24 18:27:58 +02:00
David de la Iglesia Castro 9bd9ac4a37 FileSystem: Handle encoding via kwargs. 
Drop if None.

Closes #7691
 2022-05-06 02:34:38 +03:00
David de la Iglesia Castro 8b68e4fa75 bump dvclive 0.7.3 2022-04-27 18:06:53 +05:45
David de la Iglesia Castro bea7d56bd5 plots: Fix repo with removed plots 
Changes in dvc-render 0.0.5 caused error.
 2022-04-27 11:53:30 +02:00
David de la Iglesia Castro a394498067 render.match: Skip errored revisions in get_files 2022-04-22 16:44:52 +02:00
David de la Iglesia Castro f20d6a7048 dag: Add --mermaid option. 2022-04-21 14:03:45 +05:45
David de la Iglesia Castro 5c561572b0 exp init: Replace live section with metrics and plots. 
Closes #7534
 2022-04-21 08:55:49 +02:00
David de la Iglesia Castro 179e080c2f hdfs: Fix _strip_protocol and _checksum 
Match behavior previous to #6946
 2022-04-19 16:22:49 +03:00
David de la Iglesia Castro 70b57d4703 live: Remove html generation 
Remove commands/live.
Remove LiveTask.
Bump dvclive.
 2022-04-18 15:47:31 +02:00
David de la Iglesia Castro 7dd0cffd49 gdrive: Add test_service_account_using_env_var 2022-04-08 21:39:03 +02:00
David de la Iglesia Castro 117b22d58a exp show: pcp: Allow to --drop Experiment 2022-04-04 11:48:55 +02:00
David de la Iglesia Castro ead7724824 render: vega_converter: make rev and filename optional. 
Add `FILENAME_FIELD`.
 2022-03-28 19:34:05 +02:00
David de la Iglesia Castro ddd6c97f71 render: Extract to dvc_render 
Create and use separate package for rendering logic.
Closes #6944
 2022-03-28 19:34:05 +02:00
David de la Iglesia Castro 5249207ceb live: Remove summary from tests 2022-03-07 10:09:04 +01:00
David de la Iglesia Castro 6bb3794f52 Suppress --live-no-summary help. 2022-03-02 13:30:19 +01:00
David de la Iglesia Castro f41651c97c Remove outdated SSH default jobs. 2022-02-22 12:34:25 +01:00
David de la Iglesia Castro 31a3461465 live: Update tests 2022-02-10 16:46:32 +05:45
David de la Iglesia Castro d89555cd31 exp show: handle renaming of metrics/params. 
_extend_row did not considered partially empty `items`, which can occur when metris or params are renamed between commits.

Fixes #7070
 2022-02-09 21:40:15 +01:00
David de la Iglesia Castro 37cfb8e8af exp show: Sort deps columns. 
Closes #7312
 2022-02-01 15:23:26 +01:00
David de la Iglesia Castro 8c2f8095da debug: Remove test_viztracer_internal 2022-01-27 09:36:06 +05:45
David de la Iglesia Castro 5e80fcce4f exp show: Include outs in --json output. 2022-01-26 18:45:48 +01:00
David de la Iglesia Castro 1a1e28282d exp show: Include deps columns. 
Use `repo.index.deps` to collect dependencies associated with each experiment.
 2022-01-26 18:45:48 +01:00
David de la Iglesia Castro d882e21b9b debug: Remove viztracer-depth default 2022-01-25 20:28:41 +05:45
David de la Iglesia Castro 3276c76a02 tests: Add test_debug 2022-01-25 20:28:41 +05:45
David de la Iglesia Castro f1dc9e50a7 debug: Add support for viztracer profiler. 2022-01-25 20:28:41 +05:45
David de la Iglesia Castro 822b501069 add: Update help of --jobs 2022-01-24 21:06:47 +01:00
David de la Iglesia Castro d4861d535a TabularData: Simplify drop internals. 
Update `subset` type hint.
 2022-01-22 09:41:29 +08:00
David de la Iglesia Castro 43953ace53 exp show: Add --drop and --keep args 
Removed `--include-metrics` / `--include-params` and
`--exclude-metrics` / `--exclude-params`.

Removed `--no-timestamp` . Can be done by `--drop Created`.

`--drop` and `--keep` operate directly on the table columns.

`--keep` does not perform any filtering. It's only used
to specify columns to keep despite the other filtering.

For example `--only-changed --keep foo` will prevent
`foo` from being removed by `--only-changed`.

Another example, `--drop train.* --keep train.dropout`
will remove all columns matching `train.*` except for
`train.dropout`.

Closes #7079
Closes #7080
Pre-requisite for #6434
 2022-01-20 03:28:33 +01:00
David de la Iglesia Castro 255e6a1aec TabularData: Support protected`columns. 
TabularData: Add subset arg to dropna and drop_duplicates.

Closes #7083
 2022-01-20 03:28:33 +01:00
David de la Iglesia Castro 5b8bf52eab plots: Fix --json requiring -o. 
`-o` was being required even when renderers don't need output path.
 2022-01-12 18:00:08 +01:00
David de la Iglesia Castro 0a4958bafd TabularData: Decouple pcp from render. 2022-01-03 16:30:54 +01:00
David de la Iglesia Castro 5a9f861921 exp show: Rename html to pcp 
Closes #7165
 2022-01-03 16:30:54 +01:00
David de la Iglesia Castro fc1ab7d3a1 Update show.py 2022-01-03 16:14:37 +01:00
David de la Iglesia Castro 760c8fceb3 plots: Add auto_open hint 2022-01-03 16:07:23 +01:00
David de la Iglesia Castro d8e2c14927 config: Add plots.auto_open. 
Option for enabling automatically open webbrowser with generated html.
Defaults False.

Closes #6091
 2022-01-03 16:07:23 +01:00
David de la Iglesia Castro 175ffaeb22 exp show: Remove alternating background 2022-01-03 16:38:39 +09:00
David de la Iglesia Castro 9a5614d326 TabularData: Add ignore_empty flag to drop_duplicates. 
Configures whether to consider missing values as relevant or not.
 2021-12-29 21:33:36 +01:00
David de la Iglesia Castro 12a737d905 TabularData: Fix drop_duplicates for rich_text NaN. 
When filling missing values with `ui.rich_text` (i.e. in experiments show CMD), those values were not being correctly matched against `self._fill_value`)
 2021-12-29 21:33:36 +01:00
David de la Iglesia Castro 1507a6ed99 Update benchmarks.yaml 2021-12-21 09:03:19 +05:45
David de la Iglesia Castro 4d5be46ab4 exp show --html: Handle fill_value. 
Adds new argument `fill_value` to ParallelCoordinatesRenderer.

For scalar columns, prevent converting to categorical by explictly inserting None when value is missing (value == fill_value).

For categorical columns, replace `fill_value` with `"Missing"` and ensure it is the last item so it gets rendered on top of the column.
 2021-12-17 14:45:12 +09:00
David de la Iglesia Castro 5ee82d4000 exp show: Add parallel coordinates plot. 
Uses `TabularData.to_parallel_coordinates`.
Adds new arguments: `html`, `out`, `open`

Reuses `--sort-by` to define colorscale.

Closes #4455
 2021-12-17 14:45:12 +09:00
David de la Iglesia Castro a5b7c5ea19 TabularData: Add to_parallel_coordinates. 
Uses `ParallelCoordinatesRenderer` and `dvc.render.html.write`.

pre-requisite #4455
 2021-12-17 14:45:12 +09:00
David de la Iglesia Castro 19b3c82698 dvc render: Add plotly parallel coords renderer 
New renderer based on plotly. Not exposed to `dvc plots`.
Generate plotly datapoints from `TabularData`.

pre-requisite #4455
 2021-12-17 14:45:12 +09:00
David de la Iglesia Castro 9bd60f62d7 exp show: Add alternating row colors and update border style. 
Use `box.HORIZONTALS`.

rich_table: Support `box.HORIZONTALS`

Closes #7128
 2021-12-17 14:31:53 +09:00
David de la Iglesia Castro 728731d4d7 ui: Add open_browser. 
Use `webbrowser.open` and relpath for WSL plattform.
 2021-12-05 15:44:13 +02:00
David de la Iglesia Castro fc7caa487f tests: Patch webbrowser.open on session level 
Closes #7014
 2021-11-23 22:08:21 +05:45
David de la Iglesia Castro 318b269ad1 live: Call webbrowser.open. 
Use `funcy.once_per_args` decorator to prevent opening the html multiple times if user closes first time.

Closes iterative/dvclive#180
 2021-11-22 03:35:23 +02:00
David de la Iglesia Castro 7df6090a9c params: Use ast.literal_eval for python params. 
Add unit tests for `parse_py`

Closes  #6953
 2021-11-15 17:52:49 +02:00
David de la Iglesia Castro 4a1985d7cd dvc render: Add RENDERERS registry. 
Related: #6840, #4455, #6943
 2021-11-15 16:37:24 +01:00
David de la Iglesia Castro c8903dc3a6 live: Add test for logging images with dvclive 
Closes iterative/dvclive#190
 2021-11-15 12:47:02 +01:00
David de la Iglesia Castro 28dd39a1a0 exp diff: Remove --old option. 
After #6814 the header of `diff_table` is `a_rev/`b_rev` instead of `Old`/`New` so the name of the option was counterintuitive.

In addition, make `a_rev` show by default, for consistency with output format of other `diff_table` uses (`dvc metrics diff` / `dvc params diff`).
 2021-11-12 12:58:12 +02:00
David de la Iglesia Castro 72066f233b exp show: Add --only-changed option. 
Use TabularData `drop_duplicates` to only show metrics/params with values varying across the selected experiments.

Fixes #5966
 2021-11-10 16:52:18 +02:00
David de la Iglesia Castro cb6f625dd9 TabularData: Add drop_duplicates method. 
This method mimics pandas `drop_duplicates` when axis=rows.

For axis=cols this method will drop columns with 0-variance (only
1 unique value). NaNs (`_fill_value`) won't be counted as unique values.

So the following column (`-` would be the `_fill_value`) will be dropped:

foo
-
foo
foo
 2021-11-10 16:52:18 +02:00
David de la Iglesia Castro 0638d99fe3 LiveTask: write output path at start/end 
Writing at the end of each step (create_summary) can mess up with some ML Framework progress bars, I moved the message to the monitor task, as the path won't change during training.

Fixes iterative/dvclive#177
 2021-10-29 14:41:03 +02:00
David de la Iglesia Castro 24abc6f328 TabularData: Add dropna method. 
This method mimics pandas `dropna`.
 2021-10-29 15:28:36 +03:00
David de la Iglesia Castro 4dd544121f plots: Use URI for --open. 
We were previously using relpath in *all* cases, mainly to support WSL but relpath doesn't work
on macOS.

Added a check for using relpath *only* for WSL.

Fixes #6092
 2021-10-29 13:27:12 +02:00
David de la Iglesia CastroSaugat Pachhaipre-commit-ci[bot]
 		f964a44e7c Exp show md (#6799) 
* Added `--md`/`--show-md` option to `exp show`.

* Added test_show_experiments_md

* Fixed re-defining all_experiments

* Fixed all_experiments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added dest markdown

* Added textwrap.dedent

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update experiments.py

* Apply suggestions from code review

Co-authored-by: Saugat Pachhai <[email protected]>

* Update tests/unit/command/test_compat_flag.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai <[email protected]>
 2021-10-20 11:04:57 +05:45
David de la Iglesia Castro 4f16852005 exp/metrics/params diff: Replaced Old/New with a_rev/b_rev (#6814) 
* Replaced Old/New with a_rev/b_rev

* Fixed tests
 2021-10-18 21:09:02 +03:00
David de la Iglesia Castro 7e4f443002 Added test_show_experiments_sort_by (#6798) 
* Added test_show_experiments_sort_by

* Added comment

* help formatting
 2021-10-14 14:49:00 +03:00
David de la Iglesia CastroandSaugat Pachhai 16f404b16a Fix dvclive running subdir (#6740) 
* Fix dvclive running subdir

* Apply suggestions from code review

Co-authored-by: Saugat Pachhai <[email protected]>

* Applied suggestions

* Update tests/func/test_live.py

Co-authored-by: Saugat Pachhai <[email protected]>
 2021-10-05 13:16:33 +05:45
David de la Iglesia Castro 21585d18ab Force sorted order in plots rendering (#6601) 
* Force sorted order in plots

* Fix type hint

* Fixed mypy

* Updates test_plots

* Removed reduntant test

* Removed unused import
 2021-09-16 14:25:03 +08:00
David de la Iglesia CastroJorge Orpinelpre-commit-ci[bot]
 		5c8a642500 Added GitAuthError (#6493) 
* Specify SSH protocol for `git_remote`

* Update experiments.py

* Added "a valid"

* Format

* Added GitRemoteAuthError

* Added GitRemoteAuthError

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removed unused import

* Lint

* Removed GitRemoteAuthError

* Reformatted

* Updated error message

* Use shorter title

* Fix URL

* flake8

* Flake8

* Removed extra line

* Apply suggestions from code review

Co-authored-by: Jorge Orpinel <[email protected]>

* valid -> authenticated

* Added test

* Update experiments.py

* Removed authenticated

* Minor edit

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jorge Orpinel <[email protected]>
 2021-09-13 17:56:34 +09:00
David de la Iglesia Castroandpre-commit-ci[bot] b99cd0be11 Added dvclive to tests (#6518) 
* Added dvclive to tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Moved dvclive to rest_requirements

* Update setup.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-09-03 22:50:41 +03:00
David de la Iglesia Castroandpre-commit-ci[bot] e5cf9b203d Add auto-refresh option to html (#6400) 
* Added refresh tag

* Formatted

* Added refresh_seconds to live

* Added refresh test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add refresh_seconds to render

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update html.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-09-01 17:17:23 +08:00
David de la Iglesia Castro 40abf34c8c Update dvc exp branch help (#6439) 2021-08-17 11:14:44 +09:00
David de la Iglesia CastroandPeter Rowlands c634577463 exp show: Allow for wildcard patterns in include and exclude params/m… (#5720) 
* exp show: Allow for wildcard patterns in include and exclude params/metrics

Fixes #5642

* Add test case for nested metrics/params

* match full string in _filter_name

* pre-commit

* Update dvc/command/experiments.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
 2021-04-05 20:52:29 +09:00