fix: #8612
For checkpoint experiment in some case users might want to give it a early
stopping to reduce variance. But currently if we interrupt/kill the
experiment it will be marked as failed, and all of the completed
checkpoints will be removed as we cleanup all the running directly after
the process failed.
1. We raise CheckpointKilledError intead of StageCmdFailed error if at
least one checkpoint had been commited.
2. Temp_dir executor will continue collecting data if the Checkpoint
stage was interrupted.
3. Raise warnings if a checkpoint stage was incomplete and the other
stages were not forwarded.
4. Add a new functional test for this
1. Move some basic test script from function tests to unit test.
2. Add success/failed tests for the status change of `tempdir`, `celery`, `workspace` running case.
- Overriding a config value : foo.bar=value
- Appending a config value : +foo.bar=value
- Appending or overriding a config value : ++foo.bar=value
- Removing a config value : ~foo.bar, ~foo.bar=value
See https://hydra.cc/docs/advanced/override_grammar/basic/#modifying-the-config-object
---
Breaking changes:
To modify a list, `foo[0]=bar` must now be passed as `foo.0=bar`.
Modifying a nested list inside a dictionary is not supported by omegaconf.
---
Closes#4883Closes#5868Closes#6129
* Clean up remotes's exps
fix#6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail
Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
* Push running checkpoint to remote
Fix#6182
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Get the branch name
* Finish this PR
1. move env name to dvc/env.py.
2. add some tests for it.
* Update after finally commit
* Some problems found in the review
1. change the behavior of self remote
2. do not use string DVC_EXP_AUTO_PUSH
3. downgrade the logger level in auto push
4. use full branch ref
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* add a remote check before auto push
* Split current one env into two
* Rename DVC_EXP_CHECKPOINT_PUSH to DVC_EXP_AUTO_PUSH
* value error
* Remove getenv_bool switch to env2bool
* Better on handling Dulwich exceptions
* Name changing removing prints
* Update dvc/env.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some changes in review
* Add validation to repo URL
* Add new type of SCMError
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Use spy tests call counts and args
* Update dvc/scm/git/backend/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/scm/git/backend/dulwich.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove validate_git_repo to validate_git_remote
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* live integration: get rid of dvc from live
* live: remove from api, add test for html generation during run
* stage: run: convert monitors to class context managers
* run: make monitors run in single thread
* run: move monitor logic out of run
* stage: support explicit allow_missing in checkout
* exp: explicitly clear EXEC_BRANCH and checkout with allow_missing on
starting experiment run
* git: wrap stash apply errors with SCMError
* git: add `git status` to backends
* exp: reset staged git changes before workspace exp runs to avoid merge
conflicts
* test untracked (and staged) pipeline file and deps for both workspace
and tmpdir runs
* experiments: explicitly track pipeline files and uncached deps for exp
commmits
* fix exp typing errors
* experiments: move executor into a module
* experiments: separate executor into base/local
* rename LocalExecutor to TempDirExecutor, run in .dvc/tmp by default
* git: restore HEAD using reset not checkout in detached_head()
* experiments: support calling BaseExecutor.reproduce directly for
workspace runs
* experiments: run single experiments in local workspace by default
* update experiments tests for workspace changes
* experiments: update checkpoints for workspace runs
* update checkpoints tests
* update test_show
* add force param for apply
* exp apply: add --force option
* exp run/res: add --temp option to force tempdir execution for single
experiments
* force apply in non-apply specific tests
* experiments: Allow branching from checkpoints with modified hyper params
* experiments: initial checkpoints tree functionality for `exp show`
* go back to flat/sortable view
* add test for resume w/modified params
* tests: move experiments fixtures into conftest
* add show checkpoints test
* test checkpoint table rendering
* remove dead code
* test show for checkpoint branches
pre-commit-ci[bot]andSaugat Pachhai
David de la Iglesia Castro
Peter Rowlands
skshetry
karajan1001
dependabot[bot]
Paweł Redzyński