Commit Graph
9 Commits
Author SHA1 Message Date
Peter Rowlands (변기호) f5d46c2fdf experiments: run single experiments in user's workspace by default (#5124)
* experiments: move executor into a module

* experiments: separate executor into base/local

* rename LocalExecutor to TempDirExecutor, run in .dvc/tmp by default

* git: restore HEAD using reset not checkout in detached_head()

* experiments: support calling BaseExecutor.reproduce directly for
workspace runs

* experiments: run single experiments in local workspace by default

* update experiments tests for workspace changes

* experiments: update checkpoints for workspace runs

* update checkpoints tests

* update test_show

* add force param for apply

* exp apply: add --force option

* exp run/res: add --temp option to force tempdir execution for single
experiments

* force apply in non-apply specific tests
2020-12-18 18:55:26 +09:00
Peter Rowlands (변기호) 69eb86a06f experiments: ignore untracked files (#5029)
* git: add reset()

* experiments: exclude untracked files

* experiments: add test for untracked & staged files

* git: support checkout/reset for specific paths to HEAD state

* stage: track changed files when calling checkpoint callback

* experiments: ignore untracked files when generating exp commits

* add tests for dirty lock files

* stage: track uncached outs when generating checkpoints

* experiments: update checkpoints scm handling

* update checkpoints tests
2020-12-04 17:04:45 +09:00
Peter Rowlands (변기호) 9a9f46b2b8 scm/git: add support for resolving experiment names (#5026)
* experiments: move exp ref iterators into .utils

* git: support resolving experiment short names

* update tests

* remove change unrelated to this PR
2020-12-04 15:23:08 +09:00
Ruslan Kuprieiev 5152866634 scm: move logic to gitpython backend (#5022) 2020-12-03 20:25:20 +02:00
Peter Rowlands (변기호) 176a23e291 experiments: store experiments in refs/exps instead of heads (#4910)
* experiments: commit to refs/exps/... instead of branch heads

* update exp show

* restore checkpoint_resume kwarg

* refactor new() to use experiments stash

* experiments: use refspecs to populate and collect executors

* experiments: use new stash and merge on exp checkout

* experiments: fix executor chdir

* experiments: update run and show

* skip checkout after reproduce

* checkpoints: use experiments refspecs

* check if workspace was stashed before applying

* exp show: unify checkpoint/standalone experiment collection

* git: move iter_refs to backend

* update for scm changes

* experiments: replace exp checkout in favor of exp apply

* git: handle rev_parse errors in resolve_commit

* update refs containing and baseline check

* fix sigint handling for checkpoints

* update checkpoint exists behavior

* fix checkpoint baseline behavior

* update tests

* experiments: update gc

* update show tests

* fix type checking errors

* update diff

* scm: add diff, describe

* experiments: generate human readable names

* exp run/resume: allow user specified exp name with -n/--name

* update tests

* experiments: add `exp branch` for promoting experiment to full branch

* fix test

* fix missing file

* experiments: explicitly close scm instances before cleaning up tmp dir
2020-12-02 21:20:57 +09:00
Sami Jawhar 153c374c02 experiments: pop checkpoint resume from kwargs (#4913) 2020-11-20 09:12:13 +09:00
Peter Rowlands (변기호) 379ed8e6fc experiments: support branching w/modified params from intermediate checkpoint (#4898)
* experiments: Allow branching from checkpoints with modified hyper params

* experiments: initial checkpoints tree functionality for `exp show`

* go back to flat/sortable view

* add test for resume w/modified params

* tests: move experiments fixtures into conftest

* add show checkpoints test

* test checkpoint table rendering

* remove dead code

* test show for checkpoint branches
2020-11-17 21:09:46 +09:00
Peter Rowlands (변기호) 07676b2531 checkpoints: set DVC_ROOT environment variable (#4877)
* checkpoints: set DVC_ROOT environment variable

* env: add DVC_ROOT, DVC_CHECKPOINT
2020-11-12 16:26:07 +09:00
Peter Rowlands (변기호) 389efd7ae6 checkpoints: exp run and exp res[ume] refactor (#4855)
* experiments: refactor regular/checkpoint experiment separation

* fix scm_checkout pull bug

* experiments: return stages from executor.reproduce

* since executor has to move to threadpool execution anyways, we no
  longer have to work around the pickle repro result limitation

* refactor new/resume behavior

* refactor checkpoints tests

* use -f to force checkpoints reset/restart

* checkpoints: use 'exp run' and 'exp res[ume]' commands

* stage: all outs can be missing for checkpoint stages

* regular outputs may not be written at the time a checkpoint is
  generated, but subsequent checkouts of that stage should not fail due
  to the missing outs

* commit stage on checkpoint generation otherwise non-persist cached outs
are lost

* checkpoints: wait for stage proc to end after killing via terminate()

* stage: fix issues with killing checkpoint stage

* refactor signaling (use threading.Event instead of Condition)
* fix windows bug (kill all child processes via taskkill)

* experiments: refactor executor collection

* tests: move checkpoint tests into separate module

* refactor experiment tests to use stage fixture
2020-11-11 14:19:22 +09:00