This commit adds the `key` property to the templaing context of `matrix`
stages. Similarly to `foreach` stages, the `key` is available using
`${key}` substitution.
Fixes#10043
* use popen to run updater/analytics in detached mode
subprocess does not provide a way to avoid a zombie process. Not to confuse with orphaned processes, which will be `wait`-ed by `pid 1`.
It might not work for long-running processes, but I don't think zombie processes are going to be a problem for us, as the parent process immediately exits after spawning a child process.
Even in case of `queue` based experiment runner, parent will exit before than the child process. So I don't think it's a requirement here. But it does feel wrong to leak the processes, so I have to sleep on that.
---
This PR:
1. makes `daemonized` process inherit the logging verbosity, so if the parent process used `-vv`, the daemonized process will run the same way.
2. Add `DVC_DAEMON_LOGFILE` envvar, that the daemon will log to the file. By default, the verbosity is the same as running parent processes. Without `DVC_DAEMON_LOGFILE`, the output will be redirected to `/dev/null`.
3. Adds debug/trace for updater/analytics.
I hope by doing this, we can make our life easier on debugging and make daemon more transparent/observable.
* add tests
* fixes
* use kwargs
* pass all os.environ with some modified
* exp show: show metrics that include separator
* sort using partition instead of lpartition
* handle ':' in file and metric names
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix sorting with leading :
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* update msg for multiple overlapping outputs
* update msg for multiple overlapping outputs
* fix type checking
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use sets of stages to fix mypy errors
* fix tests/func/test_add.py::test_try_adding_multiple_overlaps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] pre-commit autoupdate
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.0.282 → v0.0.285](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.282...v0.0.285)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix ruff issues
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
dberenbaum
Piotr Bielak
skshetry
Ruslan Kuprieiev
Peter Rowlands
Matt Seddon
Heinz-Alexander Fuetterer
Brady Johnson
Dave Berenbaum
David de la Iglesia Castro
Matt Seddon
