fix: #8612
For checkpoint experiment in some case users might want to give it a early
stopping to reduce variance. But currently if we interrupt/kill the
experiment it will be marked as failed, and all of the completed
checkpoints will be removed as we cleanup all the running directly after
the process failed.
1. We raise CheckpointKilledError intead of StageCmdFailed error if at
least one checkpoint had been commited.
2. Temp_dir executor will continue collecting data if the Checkpoint
stage was interrupted.
3. Raise warnings if a checkpoint stage was incomplete and the other
stages were not forwarded.
4. Add a new functional test for this
- Add `get_random_exp_name` to utils.
Generates a name (`f"{adjective}-{noun}"`) by randomly chosing pairs of adjetive, noun.
Recieves `scm` and `baseline_rev` and retries the random choice until a non-existing ref is found.
Closes https://github.com/iterative/dvc/issues/8650
Allow to include a list of potentially untracked files.
Covers the DVCLive use case where in the first run the generated files won't be tracked in Git yet.
fix: #8350
We shouldn't add locks to exp show to prevent race conditions
as this command is calling continuously by the extension.
So we make a better way to handle all kinds of reference failures.
1. Drop all based experiments if got a SCMError in baseline.
2. Drop all checkpoint experiments if any commit got a SCMError in
one of checkpoint commit.
3. Drop the entire experiments if got a SCMError.
4. Add tests for each of the conditions above.
fix: #8349
`tasks status turned to queued for 1 second before turned into success.`
is because in previous the status is using the output of
`get_running_exps` in `repo.expereiments` and when entering the
collect result progress, the `get_running_exps` will mark the tasks to
be not running because the `info.result` is not None. Here we use the
output of the celery queue as the universal standard for the tasks
status.
1. Use celery status as the exp show status
fix: #8451
1. Change the call of scm.describe from individual revision to a
collection of them. Accelerate the run speed of `exp show`
2. Bump scmrepo to 0.1.2
fix: #7986
1. Add two new flags `--hide-queued` and `--hide-failed` to `exp show`
2. Allow `exp show` to show failed experiments.
3. Add unit test for the failed experiments shown.
4. Add name support for failed exp
5. Add error msg to the `exp show` output
Currently infofile dump was spread in both executor and queue classes.
put them all into executor to make them more managable. Move all ending
dump operations to the cleanup function, because the collect_result progress
might failed because of git operations, in which can guarantee them to
be run in a `finally` scope.
1. Move some basic test script from function tests to unit test.
2. Add success/failed tests for the status change of `tempdir`, `celery`, `workspace` running case.
Reason:
1. Need some more granular control over initialization progress of
executor.
What done:
1. Seperate `init_git` and `init_cache` progress out from setup_executor
2. Move `set_ref` into `init_git`
Requires `--queue` to be used alongside `--set-param`.
Doesn't strictly require `hydra.config.enabled`.
Add `get_hydra_sweeps` to `dvc/utils/hydra.py`. Uses hydra `BasicSweeper` to generate the different sweeps.
`glob` syntax for sweeps is not supported in this P.R.
Closes#8083
---
Using setup from https://hydra.cc/docs/tutorials/basic/your_first_app/composition/ :
```console
$ dvc exp run -S "db=mysql,postgresql" -S "schema=warehouse,school" --queue
Queueing with '{'params.yaml': ['db=mysql', 'schema=warehouse']}'.
Queued experiment '5ab98b8' for future execution.
Queueing with '{'params.yaml': ['db=mysql', 'schema=school']}'.
Queued experiment '57c2fb6' for future execution.
Queueing with '{'params.yaml': ['db=postgresql', 'schema=warehouse']}'.
Queued experiment 'b9d6391' for future execution.
Queueing with '{'params.yaml': ['db=postgresql', 'schema=school']}'.
Queued experiment '145cd55' for future execution.
```
The feature will be used depending on whether `config.hydra.enabled` is True or False.
Uses `hydra.initialize_config_dir` and `hydra.compose` (from https://hydra.cc/docs/advanced/compose_api/) to build the config and dump it to `params.yaml`. The content of the output file will be overwritten.
`config.hydra.config_dir` and `config.hydra.config_name` can be used to customize the values passed to `hydra.initialize_config_dir` and `hydra.compose`.
Can be combined with `--set-param` overrides.
Closes#8082
This is a flaky, the queue tasks status might remain "Running" even if
it had already been completed, beside sometimes I also noticed it raise
a pytest fixture error.
- Overriding a config value : foo.bar=value
- Appending a config value : +foo.bar=value
- Appending or overriding a config value : ++foo.bar=value
- Removing a config value : ~foo.bar, ~foo.bar=value
See https://hydra.cc/docs/advanced/override_grammar/basic/#modifying-the-config-object
---
Breaking changes:
To modify a list, `foo[0]=bar` must now be passed as `foo.0=bar`.
Modifying a nested list inside a dictionary is not supported by omegaconf.
---
Closes#4883Closes#5868Closes#6129
Saugat Pachhai (सौगात)
dberenbaum
karajan1001
David de la Iglesia Castro
Daniele Trifirò
Ivan Shcheklein
dependabot[bot]
Peter Rowlands (변기호)
Ruslan Kuprieiev