Currently infofile dump was spread in both executor and queue classes.
put them all into executor to make them more managable. Move all ending
dump operations to the cleanup function, because the collect_result progress
might failed because of git operations, in which can guarantee them to
be run in a `finally` scope.
1. Move some basic test script from function tests to unit test.
2. Add success/failed tests for the status change of `tempdir`, `celery`, `workspace` running case.
Reason:
1. Need some more granular control over initialization progress of
executor.
What done:
1. Seperate `init_git` and `init_cache` progress out from setup_executor
2. Move `set_ref` into `init_git`
Defaults List overrides (`defaults/foo=1,2`) were not working because of the inclusion of the `/` character.
Fixed it by updating how we construct the `merged_overrides` and added a regression test.
Requires `--queue` to be used alongside `--set-param`.
Doesn't strictly require `hydra.config.enabled`.
Add `get_hydra_sweeps` to `dvc/utils/hydra.py`. Uses hydra `BasicSweeper` to generate the different sweeps.
`glob` syntax for sweeps is not supported in this P.R.
Closes#8083
---
Using setup from https://hydra.cc/docs/tutorials/basic/your_first_app/composition/ :
```console
$ dvc exp run -S "db=mysql,postgresql" -S "schema=warehouse,school" --queue
Queueing with '{'params.yaml': ['db=mysql', 'schema=warehouse']}'.
Queued experiment '5ab98b8' for future execution.
Queueing with '{'params.yaml': ['db=mysql', 'schema=school']}'.
Queued experiment '57c2fb6' for future execution.
Queueing with '{'params.yaml': ['db=postgresql', 'schema=warehouse']}'.
Queued experiment 'b9d6391' for future execution.
Queueing with '{'params.yaml': ['db=postgresql', 'schema=school']}'.
Queued experiment '145cd55' for future execution.
```
This adds support for custom metadata in outputs under `meta` key.
Also adds `--meta` flag to `add`/`import`/`import-url` that supports
`key=value` values.
preserve outputs' desc on rewrites/updates to the stage
We were not loading `desc` of the output, due to which we were not preserving the desc field over rewrites. Note that the rewrite only happens in select commands for `dvc.yaml`: `freeze`/`unfreeze`/`run`/`stage-add`, so we did not notice this before.
The feature will be used depending on whether `config.hydra.enabled` is True or False.
Uses `hydra.initialize_config_dir` and `hydra.compose` (from https://hydra.cc/docs/advanced/compose_api/) to build the config and dump it to `params.yaml`. The content of the output file will be overwritten.
`config.hydra.config_dir` and `config.hydra.config_name` can be used to customize the values passed to `hydra.initialize_config_dir` and `hydra.compose`.
Can be combined with `--set-param` overrides.
Closes#8082
We don't have aiohttp support yet, due to which we have to install
without cythonized extension.
Similarly, there has been some changes in how dataclasses disallows some
mutable arguments to be set as a class property, due to which
`hydra.conf` fails to import in 3.11.
See https://github.com/python/cpython/pull/29867. For now, I have
patched the module and avoid importing as a pytest-plugin.
dvc-hdfs/pyarrow is not there yet, we'll usually see a release 2-3
months after official reelase.
dvc-hdfs is skipped in 3.11 in extras_requires. `--set-params` will
return a nice error message if hydra cannot be imported in 3.11. I
decided to only patch that in tests.
If an untracked target is provided alongside `stages` or `deps`, don't load the parameters from the target and raise no params found instead.
Closes#8161
This was a regression introduced in #7712.
Calling `repo.metrics.show` from a subdir stopped returning metrics because `fs.isfile(metric)` check returned false.
add --no-download flag to dvc import-url/dvc update to only create/update .dvc
files without downloading the associated data.
Created .dvc files can be fetched using `dvc pull` or can be
updated using `dvc update --no-download`.
This is a flaky, the queue tasks status might remain "Running" even if
it had already been completed, beside sometimes I also noticed it raise
a pytest fixture error.
- Overriding a config value : foo.bar=value
- Appending a config value : +foo.bar=value
- Appending or overriding a config value : ++foo.bar=value
- Removing a config value : ~foo.bar, ~foo.bar=value
See https://hydra.cc/docs/advanced/override_grammar/basic/#modifying-the-config-object
---
Breaking changes:
To modify a list, `foo[0]=bar` must now be passed as `foo.0=bar`.
Modifying a nested list inside a dictionary is not supported by omegaconf.
---
Closes#4883Closes#5868Closes#6129
karajan1001
dberenbaum
Daniele Trifirò
David de la Iglesia Castro
skshetry
Ruslan Kuprieiev
dependabot[bot]
Alex Hall
Peter Rowlands (변기호)
Ivan Shcheklein