Requires `--queue` to be used alongside `--set-param`.
Doesn't strictly require `hydra.config.enabled`.
Add `get_hydra_sweeps` to `dvc/utils/hydra.py`. Uses hydra `BasicSweeper` to generate the different sweeps.
`glob` syntax for sweeps is not supported in this P.R.
Closes#8083
---
Using setup from https://hydra.cc/docs/tutorials/basic/your_first_app/composition/ :
```console
$ dvc exp run -S "db=mysql,postgresql" -S "schema=warehouse,school" --queue
Queueing with '{'params.yaml': ['db=mysql', 'schema=warehouse']}'.
Queued experiment '5ab98b8' for future execution.
Queueing with '{'params.yaml': ['db=mysql', 'schema=school']}'.
Queued experiment '57c2fb6' for future execution.
Queueing with '{'params.yaml': ['db=postgresql', 'schema=warehouse']}'.
Queued experiment 'b9d6391' for future execution.
Queueing with '{'params.yaml': ['db=postgresql', 'schema=school']}'.
Queued experiment '145cd55' for future execution.
```
This adds support for custom metadata in outputs under `meta` key.
Also adds `--meta` flag to `add`/`import`/`import-url` that supports
`key=value` values.
preserve outputs' desc on rewrites/updates to the stage
We were not loading `desc` of the output, due to which we were not preserving the desc field over rewrites. Note that the rewrite only happens in select commands for `dvc.yaml`: `freeze`/`unfreeze`/`run`/`stage-add`, so we did not notice this before.
The feature will be used depending on whether `config.hydra.enabled` is True or False.
Uses `hydra.initialize_config_dir` and `hydra.compose` (from https://hydra.cc/docs/advanced/compose_api/) to build the config and dump it to `params.yaml`. The content of the output file will be overwritten.
`config.hydra.config_dir` and `config.hydra.config_name` can be used to customize the values passed to `hydra.initialize_config_dir` and `hydra.compose`.
Can be combined with `--set-param` overrides.
Closes#8082
We don't have aiohttp support yet, due to which we have to install
without cythonized extension.
Similarly, there has been some changes in how dataclasses disallows some
mutable arguments to be set as a class property, due to which
`hydra.conf` fails to import in 3.11.
See https://github.com/python/cpython/pull/29867. For now, I have
patched the module and avoid importing as a pytest-plugin.
dvc-hdfs/pyarrow is not there yet, we'll usually see a release 2-3
months after official reelase.
dvc-hdfs is skipped in 3.11 in extras_requires. `--set-params` will
return a nice error message if hydra cannot be imported in 3.11. I
decided to only patch that in tests.
Update dvc/commands/plots.py
Co-authored-by: David de la Iglesia Castro <[email protected]>
Update tests/unit/command/test_plots.py
Co-authored-by: David de la Iglesia Castro <[email protected]>
Update dvc/commands/plots.py
Co-authored-by: Paweł Redzyński <[email protected]>
fixed test
If an untracked target is provided alongside `stages` or `deps`, don't load the parameters from the target and raise no params found instead.
Closes#8161
This was a regression introduced in #7712.
Calling `repo.metrics.show` from a subdir stopped returning metrics because `fs.isfile(metric)` check returned false.
add --no-download flag to dvc import-url/dvc update to only create/update .dvc
files without downloading the associated data.
Created .dvc files can be fetched using `dvc pull` or can be
updated using `dvc update --no-download`.
Ruslan Kuprieiev
skshetry
Daniele Trifirò
David de la Iglesia Castro
Peter Rowlands
dependabot[bot]
Dave Berenbaum
Alex Hall
Ivan Shcheklein