Commit Graph
269 Commits
Author SHA1 Message Date
Saugat Pachhai (सौगात) 896acedb07 data status: optimize printing status list
`--untracked-files` and `--unchanged` coupled with `--granular`
puts a lot of stress on `show_status`. Currently, with those options,
dvc spends ~26s on `_show_status` function.

Unhappy with the above result, I looked into improving the situation with
`console.out`. It did reduce that to 9s but it still felt like a lot.

Then I dropped to good old colorama, which dropped to 3s(term)/1.24(redirected).
I tested with buffering output which dropped this further to
~600ms(term)/~100ms(redirected).

This may seem like premature optimization, but this was preventing me
from figuring out perf issues in granular outputs, and I believe the
user should not be penalize for using feature that we already offer.
2022-07-25 17:47:23 +05:45
Saugat Pachhai (सौगात) 3dfcb82e5c add tests for data:status command 2022-07-22 16:29:04 +03:00
Saugat Pachhai (सौगात) 340a158595 introduce new dvc data status command 2022-07-22 16:29:04 +03:00
David de la Iglesia Castro c2668110b7 gc: Add --rev and --num.
Closes #7695
2022-07-12 20:13:07 +02:00
Peter Rowlands 3130a9a75f queue status: use summary style output for worker status 2022-07-12 21:39:49 +09:00
karajan1001 02681f2e65 Rename _all to all_ remove some tricky func test. 2022-07-12 21:39:49 +09:00
karajan1001 b37524603a Queue remove: Can remove successed,failed tasks msgs
Followed from #7592

1. Seperate remove method to a new file.
2. Add queued, failed, processed flags to remove.
3. Add new unit tests for queue status --queue/fail/success
4. Implement methods to remove done tasks.
5. add some new unit and functional test for celery_queue.remove
6. bump into dvc-task version 0.0.13
2022-07-12 21:39:49 +09:00
Peter Rowlands ce4cee184e exp queue: separate queue implementations into their own modules 2022-07-12 21:39:49 +09:00
karajan1001andPeter Rowlands bd74172633 queue status: add some unit test to increase the coverage
Followed by #7903

Add a new unit test to increase the coverage of test

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2022-07-12 21:39:49 +09:00
dberenbaum 164efe089f queue: ui cleanup 2022-07-12 21:39:49 +09:00
karajan1001 d7a917e2ab queue status: use a short rev to represent tasks
1. use a short rev to represent tasks
2. Add a unit test for the result shown
2022-07-12 21:39:49 +09:00
Peter Rowlands 67cbdf6de8 exp queue: remove dvc queue attach command 2022-07-12 21:39:49 +09:00
Peter Rowlands f12091002f exp queue: add dvc queue logs command 2022-07-12 21:39:49 +09:00
karajan1001 3d0ee0d202 exp queue: monitor queue status (#7590)
fix: #7590
1. Add two new command `queue status` and `queue attach`
2. Add unit test for the CLI
3. Make `Running` exps can be found by `get_queue_entry_by_names`
2022-07-12 21:39:49 +09:00
karajan1001 12fc8c4e7d exp queue: Add two command queue start and stop (#7589)
fix: #7589

1. Add two sub-command `dvc queue start` and `dvc queue stop`
2. Add a unit test to test them
2022-07-12 21:39:49 +09:00
karajan1001andPeter Rowlands a6cb4eb72e exp queue: remove queued experiments(#7591)
fix: #7591
1. Add a new command `dvc queue`
2. Add two sub-command `dvc queue remove` and `dvc queue kill`
3. Add a unit test to test them

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2022-07-12 21:39:49 +09:00
15ada9a1c3 Implement --date to exp gc
1. Implement `--time` to `exp gc`
2. add a unit test for it.
3. Rename `--time` to `--date` and change the warning msg.
4. To get a better msg for `gc` and `exp gc`

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: Dave Berenbaum <[email protected]>
2022-07-12 13:53:46 +02:00
karajan1001andDave Berenbaum 20d1f7a87d gc: add a new flag --time (#678)
fix #678
1. Add a new flag `--time` to `dvc gc`.
2. Modify the unit test of `dvc gc`
3. Fix an argument checking error and update the error msg.

Co-authored-by: Dave Berenbaum <[email protected]>
2022-07-12 13:53:46 +02:00
karajan1001 9c59689aa7 Gc: Move old unittest gc tests to pytest 2022-07-12 13:53:46 +02:00
Paweł Redzyński fbb98a87d1 plots: introduce flexible plots configuration to dvcfiles
Closes: #7086
2022-07-01 13:17:24 +02:00
Saugat Pachhai (सौगात) 4bc6b5699a quote node names 2022-06-15 12:43:45 +03:00
dberenbaum 33b3afa2f5 exp init: create output dirs 2022-06-07 17:55:24 +05:45
David de la Iglesia Castro 3d791778de plots: Pass templates_dir to match_renderers.
We were forgetting to pass default template dir.

Fixes #7817.
2022-05-31 16:17:58 +02:00
Ruslan Kuprieiev 844f77b4f9 dvc: drop python 3.7
Also upgraded with `pyupgrade --py38-plus`

Fixes #7708
2022-05-30 16:51:35 +03:00
David de la Iglesia Castro 109d0e4e28 dag: mermaid: Use quotation marks to escape node name.
Fixes #7731
2022-05-24 18:27:58 +02:00
Karthikeyan SingaravelanandDavid de la Iglesia Castro 3faf413288 Use sorted output in test since order is not deterministic.
Co-authored-by: David de la Iglesia Castro <[email protected]>
2022-05-11 18:40:21 +03:00
Karthikeyan Singaravelan a3261637de dag: fix dot file rendering order. 2022-05-11 18:40:21 +03:00
dberenbaum 477f266dc5 exp init: fixes #7534; simplifies/updates exp init --live 2022-05-09 20:58:27 +05:45
David de la Iglesia Castro f20d6a7048 dag: Add --mermaid option. 2022-04-21 14:03:45 +05:45
Peter Rowlands 444bf27e6f exp show: add internal --no-fetch option to disable temp/queue ref fetch 2022-04-21 15:45:59 +09:00
Saugat Pachhai (सौगात) 02b2ab6804 stage/exp list: alias --names-only to --name-only
Fixes #7570.
2022-04-21 06:53:16 +05:45
David de la Iglesia Castro 70b57d4703 live: Remove html generation
Remove commands/live.
Remove LiveTask.
Bump dvclive.
2022-04-18 15:47:31 +02:00
Paweł Redzyński 03f553c54b plots: json: show even if no results
Related: iterative/vscode-dvc#1521
2022-04-07 11:45:01 +02:00
Saugat Pachhai (सौगात) 7d0d6a38d7 exp init: change ui prompts structure and output 2022-03-29 16:20:32 +05:45
David de la Iglesia Castro ead7724824 render: vega_converter: make rev and filename optional.
Add `FILENAME_FIELD`.
2022-03-28 19:34:05 +02:00
David de la Iglesia Castro ddd6c97f71 render: Extract to dvc_render
Create and use separate package for rendering logic.
Closes #6944
2022-03-28 19:34:05 +02:00
karajan1001andJorge Orpinel efc021da73 exp remove: rename --all-commits flag and unify the collection of revs (#7155)
fix: #7155
1. rename flags `-A/--all-commits` in `exp remove`
2. add new flag `-n/--num` in `exp remove`
3. unify the revision collection in `exp remove`
4. add unit and func tests for `exp remove`

Co-authored-by: Jorge Orpinel <[email protected]>
2022-03-09 18:54:15 +08:00
karajan1001 244637e299 exp push/pull: add --all-commits flag and unify the collection of revs (#7154)
fix: #7154
1. rename flags `-A/--all-commits` in the `exp pull/push`
2. add new flag "--rev" and "--num" in the `exp pull/push`
3. Unify the collection of revs in `exp push/pull`
4. add unit and func tests for `exp push/pull`
2022-03-09 17:43:26 +08:00
karajan1001andJorge Orpinel d033345860 exp list: add the new flags (#7153)
fix: #7153
Unify the flags in `exp list` to `exp show`

1. Add new `-A`, `-n` flags to `exp list` command.
2. Add unit test for the flag parse.
3. unify the revs retrive logic to `exp show`.
4. Add functional test for `exp list`
5. Update the documents.

Co-authored-by: Jorge Orpinel <[email protected]>
2022-03-08 16:17:33 +08:00
Peter Rowlands d80c43135f exp run: remove unsupported (repro only) arguments 2022-03-07 12:01:51 +01:00
dberenbaum fc3f3ca8d7 diff: fix missing key bug (#6720) 2022-03-02 19:10:31 +01:00
Paweł Redzyński 3b4d76df42 plots: json: add datapoints to vega plots 2022-02-25 17:12:45 +01:00
Paweł Redzyński 7bc14a5ddb plots: make templates generation explicit
Fixes: #6389
2022-02-01 15:09:25 +01:00
David de la Iglesia Castro 8c2f8095da debug: Remove test_viztracer_internal 2022-01-27 09:36:06 +05:45
David de la Iglesia Castro 3276c76a02 tests: Add test_debug 2022-01-25 20:28:41 +05:45
David de la Iglesia Castro d4861d535a TabularData: Simplify drop internals.
Update `subset` type hint.
2022-01-22 09:41:29 +08:00
David de la Iglesia Castro 43953ace53 exp show: Add --drop and --keep args
Removed `--include-metrics` / `--include-params` and
`--exclude-metrics` / `--exclude-params`.

Removed `--no-timestamp` . Can be done by `--drop Created`.

`--drop` and `--keep` operate directly on the table columns.

`--keep` does not perform any filtering. It's only used
to specify columns to keep despite the other filtering.

For example `--only-changed --keep foo` will prevent
`foo` from being removed by `--only-changed`.

Another example, `--drop train.* --keep train.dropout`
will remove all columns matching `train.*` except for
`train.dropout`.

Closes #7079
Closes #7080
Pre-requisite for #6434
2022-01-20 03:28:33 +01:00
Saugat Pachhai (सौगात) 446516d788 refactor: rename dvc.command to dvc.commands 2022-01-19 19:41:40 +05:45
Saugat Pachhai (सौगात) 4268b5a339 cli: don't inherit DvcParserError from DvcException, move to dvc.cli 2022-01-19 19:41:40 +05:45
Saugat Pachhai (सौगात) f5baf990cf cli: refactor into a module
By splitting dvc.cli into a module, we'll be able to share cli-related
utilities, split parser-related stuffs into smaller parts, improve
reusability and improve API for command extensions in the future.
2022-01-19 12:22:07 +02:00