The code are wrapped due to us using line-length=79 before, but now some
of the code can easily fit in 88 chars.
The formatter did not change this automatically when we changed line-length,
because of magic trailing commas that gets added when they turn into multiple lines.
Black/ruff wrapping them in parens and splitting them into multiple lines
was making it difficult to read.
* drop Python 3.8
Note that this PR only makes typing related changes.
Other changes can be done once this gets merged, they
are relatively few in numbers.
* bump build python version
* add typing_extensions to lint group
* repro: introduce --keep-going/--ignore to continue on error
`-k`/`--keep-going` will skip all dependents of a stage if it fails.
It'll still try to reproduce the other dependencies of the pending
targets, before it exits with a non-zero status.
`--ignore-errors` will treat any error during stage execution as a
success, and will continue executing other dependencies as if it
succeeded. Exits with 0 even on failure.
Also fixes:
* --force-downstream used to force everything even if one upstream nodes changed.
This used to work when we supported a single targets, but there could be independent
pipelines, so we should not force everything if any node changes.
Now we'll force reproduce the stage only if one of its dependency changed upstream
or it's upstream node was also forced.
* Makes `--downstream` more correct, in relation to `--allow-missing` and `--pull`.
Now we use an active graph for downstream/upstream logic instead of a subgraph.
Note that this does not affect the downstream logic though, as dvc always goes
downstream, so subgraph was no different than active graph.
The subgraph only did not have the upstream nodes.
* add tests
* add spacing in the message
The logic is now moved to experiments, where it tracks all the stage related files
itself. The `_get_stage_files()` has been modified to also return lockfile path,
so that it can be staged, and has been moved to `stage/utils.py`, and out of `reproduce.py`.
Fixes#9688.
* repro: make stages run in order of the graph.
In case of --single-item, this means it'll run in the order of arguments passed.
For --downstream, this fixes running of stage out of order when multiple arguments are passed.
For normal case, this enforces the same logic, so that they are never run out-of-order.
* add tests
* tests: repro: add test for --single-item with multiple targets
* graph: support when no nodes are provided
Had the "pleasure" of finding a mocker.patch leak into other tests and one
of the suspicions is that mixing unittest.mock and mocker might be somehow
related.
In any case, we should use mocker where possible anyway.
* cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc.
* remove pylint direcitves
* Apply suggestions from code review
* Update pyproject.toml
fix: #8624
1. Add a new flag `--force` for `queue kill`
2. Make `SIGINT` to be the default option and `SIGKILL` to be with
`--force`
3. Add tests for `queue kill`
4. Bump dvc-task into 0.1.9
fix: #8612
For checkpoint experiment in some case users might want to give it a early
stopping to reduce variance. But currently if we interrupt/kill the
experiment it will be marked as failed, and all of the completed
checkpoints will be removed as we cleanup all the running directly after
the process failed.
1. We raise CheckpointKilledError intead of StageCmdFailed error if at
least one checkpoint had been commited.
2. Temp_dir executor will continue collecting data if the Checkpoint
stage was interrupted.
3. Raise warnings if a checkpoint stage was incomplete and the other
stages were not forwarded.
4. Add a new functional test for this
fix: #8702
1. replace simple `proc.kill` in `queue stop --kill` with the same logic as in
`queue kill` to make it possible to set false status `running` tasks
as faileed.
2. Modified the tests.
skshetry
pre-commit-ci[bot]andSaugat Pachhai
Matt Seddon
Matt Seddon
Heinz-Alexander Fuetterer
dberenbaum
Oded Messer
Ruslan Kuprieiev
Peter Rowlands
David de la Iglesia Castro
karajan1001