Commit Graph
246 Commits
Author SHA1 Message Date
skshetry 93eea0cd42 deps: bump scmrepo to 0.1.15; close subrepo.dvc on tests teardown (#9122) 2023-03-06 20:21:42 +05:45
Peter Rowlands 2cf3a12e46 tests: unmark exp celery tests as xfail 2023-03-02 15:44:34 +09:00
Peter Rowlands 78fb5275e5 exp: add dvc exp clean command 2023-03-02 12:35:54 +09:00
Peter Rowlands ad5b91e51a exp show: cache collected git revs 2023-02-24 14:18:58 +09:00
David de la Iglesia Castro 3043b6588f exp save: Use pathspec to handle include_untracked.
Closes #9061
2023-02-21 09:43:25 +01:00
Peter Rowlands 648edd71eb exp: move apply tests into their own module 2023-02-20 14:36:38 +09:00
Peter Rowlands 0c95cde712 exp: remove unused ApplyConflictError 2023-02-20 14:36:38 +09:00
skshetry 1de37556d5 bump ruff to v0.0.247 (#9050)
* bump ruff to v0.0.247

* ruff: enable pep8-naming check

rename some camelCased APIs, especially in logger
rename G variables to graph
ignore some variable namings and help messages in dvc.commands
2023-02-18 16:50:35 +05:45
David de la Iglesia Castro b7b7eff4a3 tests: Fix test_deleted.
Closes #6297
#9017 actually fixed this but the test was incorrect (something wrong with caplog).
2023-02-16 11:43:09 +01:00
David de la Iglesia Castro 320771551b exp: Preserve staged changes after experiments.
- Detach head after stashing workspace.
- Use reinstate_index.
2023-02-14 16:59:10 +01:00
David de la Iglesia Castro 1676d1f344 Add test_stash_exp. 2023-02-14 16:59:10 +01:00
Ruslan Kuprieiev 58295ca0ab repo: rename odb -> cache
We used to have it named `cache` before, but these days it is clear that
you can have three odbs: just odb (e.g. in-memory), cache (typically local)
or remote (typically cloud). Fixing naming here before proceeding with
introducing `out.odb` and `out.cache`.

In preparation for migration to https://github.com/iterative/dvc-data/pull/304
2023-02-08 23:05:45 +02:00
Hagai Kariti 6c8c26f741 Support setting a exp name prefix in hydra sweeps 2023-02-07 18:49:23 +01:00
d84196db62 studio: Move post_live_metrics from queue to executor (#8955)
* studio: Move `post_live_metrics` from `queue/workspace` to `executor/base`.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update dvc/repo/experiments/utils.py

Co-authored-by: skshetry <[email protected]>

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: skshetry <[email protected]>
2023-02-06 13:31:10 +00:00
skshetry a86470ccbc update black and change line-length to 88 (#8953) 2023-02-04 05:28:07 +02:00
98bcfad8e7 build(deps-dev): bump scmrepo from 0.1.6 to 0.1.7 (#8821)
* build(deps-dev): bump scmrepo from 0.1.6 to 0.1.7

Bumps [scmrepo](https://github.com/iterative/scmrepo) from 0.1.6 to 0.1.7.
- [Release notes](https://github.com/iterative/scmrepo/releases)
- [Commits](https://github.com/iterative/scmrepo/compare/0.1.6...0.1.7)

---
updated-dependencies:
- dependency-name: scmrepo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* mypy: remove scmrepo from ignored modules

* Fix mypy issues

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: skshetry <[email protected]>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-01-28 14:26:00 +05:45
skshetry 7ab4c59d70 test: run pytest-style checks (#8901) 2023-01-28 14:10:53 +05:45
David de la Iglesia Castro 9af6fcad4a exp run: Fix --dry option.
Closes #8831
2023-01-17 13:32:20 +01:00
skshetry c47c2123fb cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc (#8822)
* cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc.

* remove pylint direcitves

* Apply suggestions from code review

* Update pyproject.toml
2023-01-17 08:35:19 +05:45
skshetry b0052aa666 pre-commit update: flake8/ruff/pyupgrade/black/isort (#8813) 2023-01-15 14:56:54 +05:45
skshetry da27174e02 dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups, etc. (#8812)
dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups etc
2023-01-15 12:45:10 +05:45
karajan1001 fc1a192c90 Bump pathspec to 0.10.3
fix: #8217
1. Replace group name to avoid conflict.
2. Disable failed tests.
3. Bump pathspec to 0.10.3
2023-01-13 22:47:36 +08:00
dependabot[bot]andSaugat Pachhai 178244f666 build(deps-dev): bump dvc-task from 0.1.9 to 0.1.10 (#8807)
* build(deps-dev): bump dvc-task from 0.1.9 to 0.1.10

Bumps [dvc-task](https://github.com/iterative/dvc-task) from 0.1.9 to 0.1.10.
- [Release notes](https://github.com/iterative/dvc-task/releases)
- [Commits](https://github.com/iterative/dvc-task/compare/0.1.9...0.1.10)

---
updated-dependencies:
- dependency-name: dvc-task
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* remove xfail on json.JSONDecodeError/PermissionError

JSONDecodeError should be fixed by the new release. Next we need to investigate
PermissionError.

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-01-13 17:25:36 +05:45
skshetry a814f04ee0 tests: xfail test_apply_failed on JSONDecodeError/PermissionError (#8796) 2023-01-11 18:04:23 +05:45
skshetry 90a27179eb ruff: ignore complex functions and remove dvc.utils.fs.makedirs (#8795)
We may want to prevent any new functions from getting complex.
2023-01-11 17:58:51 +05:45
skshetry 175c578579 ruff: ban relative parent/grandparent imports (#8794) 2023-01-11 16:48:46 +05:45
skshetry 5441964fdb add ruff (#8793)
* add ruff linter

* enable few more checks
2023-01-11 15:20:38 +05:45
Saugat Pachhai (सौगात) 3dfdd577e7 tests: xfail experiments-related flakey tests 2023-01-06 18:59:53 +05:45
dberenbaum 6ac7dfc008 exp: fix tests and remove reset checks 2023-01-04 17:29:12 +01:00
dberenbaum b0372918ea exp: stop checking for unchanged experiments 2023-01-04 17:29:12 +01:00
karajan1001 f386f7c082 exp apply can accept exp names and failed exps as target
fix: #8182

1. `exp apply` can apply to failed experiments.
2. `exp apply` can accept exp names as target.
3. Add a new functional test for it.
2022-12-30 17:40:18 +08:00
David de la Iglesia Castro 68b24307f9 post_live_metrics: Pass params on start event. 2022-12-28 11:43:37 +01:00
karajan1001 08ae68caf4 Make temp dir executed checkpoint experiment return result to workspace.
fix: #8612
For checkpoint experiment in some case users might want to give it a early
stopping to reduce variance.  But currently if we interrupt/kill the
experiment it will be marked as failed, and all of the completed
checkpoints will be removed as we cleanup all the running directly after
the process failed.

1. We raise CheckpointKilledError intead of StageCmdFailed error if at
   least one checkpoint had been commited.
2. Temp_dir executor will continue collecting data if the Checkpoint
   stage was interrupted.
3. Raise warnings if a checkpoint stage was incomplete and the other
   stages were not forwarded.
4. Add a new functional test for this
2022-12-26 18:25:43 +05:45
David de la Iglesia Castro b13184fcec tests: Fix flaky test_show_only_changed.
Some of the random exp names could start with `bar`.

https://github.com/iterative/dvc/actions/runs/3757008396/jobs/6383731723#step:5:79
2022-12-22 13:01:46 +01:00
David de la Iglesia Castro d7fd68a1ae Add start and done calls with post_live_metrics. 2022-12-22 11:33:59 +01:00
Daniele Trifirò 59d6d5ea98 exp save: simplify implementation 2022-12-22 11:01:41 +01:00
David de la Iglesia Castro bb7512687c exp: Generate a human-readable name beforehand.
- Add `get_random_exp_name` to utils.
Generates a name (`f"{adjective}-{noun}"`) by randomly chosing pairs of adjetive, noun.
Recieves `scm` and `baseline_rev` and retries the random choice until a non-existing ref is found.

Closes https://github.com/iterative/dvc/issues/8650
2022-12-09 10:32:38 +01:00
Saugat Pachhai (सौगात) 334f297ad3 deps: remove 3.11 checks for hydra; has 3.11 support now 2022-12-09 10:32:12 +05:45
David de la Iglesia Castro a036eaf817 exp: save: Add include_untracked.
Allow to include a list of potentially untracked files.
Covers the DVCLive use case where in the first run the generated files won't be tracked in Git yet.
2022-11-28 15:49:34 +01:00
David de la Iglesia Castro 665185b9e7 exp: save: Handle staged changes.
Staged changes were causing a merge error.
Warn and unstage instead of erroring, to match exp run behavior.
2022-11-28 15:49:34 +01:00
David de la Iglesia Castro 43e281521e exp save: cleanup.
Remove metrics.

Move tests to separate file.

Fix `experiments.get_exact_name` usage.

Remove unused checkpoint logic.
2022-11-28 15:49:34 +01:00
Daniele Trifirò 73b46a5789 exp save: initial implementation 2022-11-28 15:49:34 +01:00
David de la Iglesia Castro 8124396218 exp: Expose baseline and name via run_env.
Required by DVCLive for the experiments onboarding and live metrics features.

Related (not sure if closes): https://github.com/iterative/dvc/issues/6668
2022-11-28 13:13:04 +01:00
karajan1001 6bfce5a5db Use a more granular lock on exp operations.
For now in `experiment.new`, `init_executor` and `collect_exectuor` we
lock the whole scm repo. We can make a more granular lock in these
operations.
2022-11-28 17:04:16 +08:00
karajan1001 5fa33a06bc Move EXEC_TMP_DIR and EXEC_PID_DIR to experiment.utils 2022-11-28 17:04:16 +08:00
Ivan Shcheklein d7587556c9 exp list: cleanup and move logic inside repo api 2022-11-25 16:15:17 -08:00
Ivan Shcheklein 76cc649f5e fix/8550+: cover with test 2022-11-16 15:16:37 +05:45
David de la Iglesia Castro e4ea463c73 hydra: Raise error when name and sweeps. 2022-11-14 21:34:55 +01:00
karajan1001 fa54c1af7c Solve run_all flaky tests
1. Modify run all to include currently running exps.
2. bump dvc-task to 0.1.5
2022-11-11 07:04:43 +05:45
karajan1001 9fbbfd07a6 Fix some celery queue realted ci failure
fix: #8403
1. remove some of the flaky mark
2. In `get_result` make sure the celery task is completed.
2022-11-11 07:04:43 +05:45