Commit Graph
4 Commits
Author SHA1 Message Date
Sami Jawhar 153c374c02 experiments: pop checkpoint resume from kwargs (#4913) 2020-11-20 09:12:13 +09:00
Peter Rowlands (변기호) 379ed8e6fc experiments: support branching w/modified params from intermediate checkpoint (#4898)
* experiments: Allow branching from checkpoints with modified hyper params

* experiments: initial checkpoints tree functionality for `exp show`

* go back to flat/sortable view

* add test for resume w/modified params

* tests: move experiments fixtures into conftest

* add show checkpoints test

* test checkpoint table rendering

* remove dead code

* test show for checkpoint branches
2020-11-17 21:09:46 +09:00
Peter Rowlands (변기호) 07676b2531 checkpoints: set DVC_ROOT environment variable (#4877)
* checkpoints: set DVC_ROOT environment variable

* env: add DVC_ROOT, DVC_CHECKPOINT
2020-11-12 16:26:07 +09:00
Peter Rowlands (변기호) 389efd7ae6 checkpoints: exp run and exp res[ume] refactor (#4855)
* experiments: refactor regular/checkpoint experiment separation

* fix scm_checkout pull bug

* experiments: return stages from executor.reproduce

* since executor has to move to threadpool execution anyways, we no
  longer have to work around the pickle repro result limitation

* refactor new/resume behavior

* refactor checkpoints tests

* use -f to force checkpoints reset/restart

* checkpoints: use 'exp run' and 'exp res[ume]' commands

* stage: all outs can be missing for checkpoint stages

* regular outputs may not be written at the time a checkpoint is
  generated, but subsequent checkouts of that stage should not fail due
  to the missing outs

* commit stage on checkpoint generation otherwise non-persist cached outs
are lost

* checkpoints: wait for stage proc to end after killing via terminate()

* stage: fix issues with killing checkpoint stage

* refactor signaling (use threading.Event instead of Condition)
* fix windows bug (kill all child processes via taskkill)

* experiments: refactor executor collection

* tests: move checkpoint tests into separate module

* refactor experiment tests to use stage fixture
2020-11-11 14:19:22 +09:00