The code are wrapped due to us using line-length=79 before, but now some
of the code can easily fit in 88 chars.
The formatter did not change this automatically when we changed line-length,
because of magic trailing commas that gets added when they turn into multiple lines.
Black/ruff wrapping them in parens and splitting them into multiple lines
was making it difficult to read.
* drop Python 3.8
Note that this PR only makes typing related changes.
Other changes can be done once this gets merged, they
are relatively few in numbers.
* bump build python version
* add typing_extensions to lint group
Had the "pleasure" of finding a mocker.patch leak into other tests and one
of the suspicions is that mixing unittest.mock and mocker might be somehow
related.
In any case, we should use mocker where possible anyway.
* cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc.
* remove pylint direcitves
* Apply suggestions from code review
* Update pyproject.toml
fix: #8624
1. Add a new flag `--force` for `queue kill`
2. Make `SIGINT` to be the default option and `SIGKILL` to be with
`--force`
3. Add tests for `queue kill`
4. Bump dvc-task into 0.1.9
fix: #8612
For checkpoint experiment in some case users might want to give it a early
stopping to reduce variance. But currently if we interrupt/kill the
experiment it will be marked as failed, and all of the completed
checkpoints will be removed as we cleanup all the running directly after
the process failed.
1. We raise CheckpointKilledError intead of StageCmdFailed error if at
least one checkpoint had been commited.
2. Temp_dir executor will continue collecting data if the Checkpoint
stage was interrupted.
3. Raise warnings if a checkpoint stage was incomplete and the other
stages were not forwarded.
4. Add a new functional test for this
fix: #8702
1. replace simple `proc.kill` in `queue stop --kill` with the same logic as in
`queue kill` to make it possible to set false status `running` tasks
as faileed.
2. Modified the tests.
* re-enable --jobs concurrency
* use `start_workers` instead of `spawn_worker`
* make `spawn_worker` a private method.
* update tests: make `spawn_worker` a private method.
fix#7693
The problem was when we gathered experiments, and at the same time,
the executor are creating new commits. It will make the checkpoint
top/head return by `get_running_exps`/`get_ref`/`branch_revs` not the
same one. And cause the problem described in #7693
To solve this we did
1. move the `get_rev` to the beginning of `exp show` to make sure the
return results in `branch_revs`
2. Add a new function to move all of the properties that only belong to
`head` of the checkpoint from the wrong place to the top.
3. Add a new unit test for it.
4. Force fetch_running_exp to return data.
5. fix a bug in tmp_dir_executor.
fix: #8461fix: #8473
For now, for those inactive but uncompleted tasks, they still shown as
Running in `exp show` and `task queue`, we cannot remove them because
they are regarded as running, we cann't kill them because their are no
real process behind them.
1. Now we can examine these false running tasks and mark them as failure
in `queue kill` command. #8461
2. Better msg handle for initliaizing tasks in `queue kill`#8473
3. Add new unit tests for both cases.
4. bump into new dvc-task 0.1.6
fix: #8526
1. The current `get_names` will crash if an experiment branch had
already been given a new branch name to it.
1. Solve the crash when we getting the name of a newly fixed experiment
branch.
2. Add a new unit test to it.
Currently infofile dump was spread in both executor and queue classes.
put them all into executor to make them more managable. Move all ending
dump operations to the cleanup function, because the collect_result progress
might failed because of git operations, in which can guarantee them to
be run in a `finally` scope.
1. Move some basic test script from function tests to unit test.
2. Add success/failed tests for the status change of `tempdir`, `celery`, `workspace` running case.
Reason:
1. Need some more granular control over initialization progress of
executor.
What done:
1. Seperate `init_git` and `init_cache` progress out from setup_executor
2. Move `set_ref` into `init_git`
fix: #8014
> ERROR: Invalid experiment '{entry.stash_rev[:7]}'.
This happens when the queue task failed from a scm error.
{"exc_type": "GitMergeError", "exc_message": ["Cannot fast-forward HEAD
to '05100047a341f2fa4a02421289d48f84d8c45e86'"], "exc_module": "dvc.scm"}.
And the failed task neither create a infofile nor create a fail_stash.
1. Didn't raise DvcException if no infofile found for failed tasks.
(success tasks still raise it)
2. Add a new unit test for this.
fix: #8018fix: #8071
1. Add one line to make `exp remove` can remove failed tasks, and will
also delete associated successful tasks
2. Add a new unit test for this behaviour.
3. Solve the unstable test_queue problem.
fix: #8031
Currently when we use tmp directory to run a experiment, we set
`EXEC_HEAD` , `EXEC_BASELINE` and `EXEC_MERGE` heads locally but forget
to remove it. This cause some problem when we use `exp show`
1. Clean `EXEC_HEAD` , `EXEC_BASELINE` and `EXEC_MERGE` after we pushed
them to tmp_dir
2. Add a unit test for it.
Co-authored-by:
skshetry
pre-commit-ci[bot]andSaugat Pachhai
Heinz-Alexander Fuetterer
dberenbaum
Oded Messer
Ruslan Kuprieiev
David de la Iglesia Castro
Peter Rowlands
karajan1001
Fabian Zills
Daniele Trifirò