Cuts down `dvc ls . data/mnist/dataset --recursive` for me from ~37 to ~25
seconds.
Note that this also changes behaviour for ls with a broken symlink, as we
were not able to see it before because of a failing `info()` call, but now
we can follow a normal UNIX ls behaviour more closely.
fix: #8624
1. Add a new flag `--force` for `queue kill`
2. Make `SIGINT` to be the default option and `SIGKILL` to be with
`--force`
3. Add tests for `queue kill`
4. Bump dvc-task into 0.1.9
fix: #8612
For checkpoint experiment in some case users might want to give it a early
stopping to reduce variance. But currently if we interrupt/kill the
experiment it will be marked as failed, and all of the completed
checkpoints will be removed as we cleanup all the running directly after
the process failed.
1. We raise CheckpointKilledError intead of StageCmdFailed error if at
least one checkpoint had been commited.
2. Temp_dir executor will continue collecting data if the Checkpoint
stage was interrupted.
3. Raise warnings if a checkpoint stage was incomplete and the other
stages were not forwarded.
4. Add a new functional test for this
fix: #8702
1. replace simple `proc.kill` in `queue stop --kill` with the same logic as in
`queue kill` to make it possible to set false status `running` tasks
as faileed.
2. Modified the tests.
* re-enable --jobs concurrency
* use `start_workers` instead of `spawn_worker`
* make `spawn_worker` a private method.
* update tests: make `spawn_worker` a private method.
- Add `get_random_exp_name` to utils.
Generates a name (`f"{adjective}-{noun}"`) by randomly chosing pairs of adjetive, noun.
Recieves `scm` and `baseline_rev` and retries the random choice until a non-existing ref is found.
Closes https://github.com/iterative/dvc/issues/8650
Allow to include a list of potentially untracked files.
Covers the DVCLive use case where in the first run the generated files won't be tracked in Git yet.
* ui: Fix WSL check in `open_browser`
* Update dvc/ui/__init__.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
fix#7693
The problem was when we gathered experiments, and at the same time,
the executor are creating new commits. It will make the checkpoint
top/head return by `get_running_exps`/`get_ref`/`branch_revs` not the
same one. And cause the problem described in #7693
To solve this we did
1. move the `get_rev` to the beginning of `exp show` to make sure the
return results in `branch_revs`
2. Add a new function to move all of the properties that only belong to
`head` of the checkpoint from the wrong place to the top.
3. Add a new unit test for it.
4. Force fetch_running_exp to return data.
5. fix a bug in tmp_dir_executor.
fix: #8461fix: #8473
For now, for those inactive but uncompleted tasks, they still shown as
Running in `exp show` and `task queue`, we cannot remove them because
they are regarded as running, we cann't kill them because their are no
real process behind them.
1. Now we can examine these false running tasks and mark them as failure
in `queue kill` command. #8461
2. Better msg handle for initliaizing tasks in `queue kill`#8473
3. Add new unit tests for both cases.
4. bump into new dvc-task 0.1.6
* hydra: Use `OmegaConf.to_yaml` for dumping `.yaml` output.
Combining `OmegaConf.to_object` with our yaml DUMPER (ruaml.yaml) was not correctly handling strings like 'no'.
`OmegaConf.to_yaml` uses a custom string representer that correctly handles those cases.
Fixes#8583
* Update dvc/utils/hydra.py
Co-authored-by: Dave Berenbaum <[email protected]>
Co-authored-by: Dave Berenbaum <[email protected]>
skshetry
Ivan Shcheklein
dberenbaum
David de la Iglesia Castro
Ruslan Kuprieiev
Saugat Pachhai (सौगात)
Daniele Trifirò
karajan1001
Fabian Zills
Yury
