Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,746 Commits 7 Branches 579 Tags
a1b891ce01c68c1f4cd891b8f72188a321bd75a5
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
David de la Iglesia Castro a1b891ce01 deps: bumps dvc-azure>=2.21.2 
Closes #9506 via https://github.com/iterative/dvc-azure/pull/48
 2023-05-31 14:22:27 +02:00
daavoo 5a6f834088 repro: Skip pulling run cache if no-run-cache was passed. 2023-05-25 16:35:30 +02:00
daavoo e98bf38d4c repro: Continue on RunCacheNotSupported 
Closes #9470
 2023-05-25 16:35:30 +02:00
David de la Iglesia Castro 25313f5d82 exp: Drop checkpoints (#9271) 
exp: Drop `checkpoints`.

Closes #9221
 2023-05-23 11:32:18 +05:45
David de la Iglesia Castro 182728d674 exp run / repro: Drop --metrics. (#9458) 
* exp run / repro: Drop `--metrics`.

* exp run: Rename `-M` to `-m`.
 2023-05-22 19:13:22 +05:45
David de la Iglesia Castro f245567613 exp: Drop gc. (#9273) 
Closes #7991
 2023-05-22 16:08:25 +05:45
daavooandIvan Shcheklein eb45cda5d0 get_random_exp_name: Use custom random generator. 
See https://github.com/iterative/dvclive/pull/576

Co-authored-by: Ivan Shcheklein <[email protected]>
 2023-05-22 10:46:41 +02:00
daavoo 3ddd4b87fc exp: Fix --rev args for ls pull and remove. 
Closes #9471
 2023-05-21 13:25:40 +02:00
David de la Iglesia Castro 02f7fcae73 exp show: Drop --pcp. (#9270) 
Closes #9268
 2023-05-19 15:36:33 +05:45
David de la Iglesia Castro cdba65f27e exp: Drop init. (#9269) 
Drop `exp init`.

Closes #9267
 2023-05-19 14:39:16 +05:45
daavoo ecb71c8263 exp push: Handle rev arg as list. 2023-05-18 09:54:10 +02:00
David de la Iglesia Castro e6e3912da0 testing: Lazy import pytest_virtualenv (#9469) 2023-05-17 16:27:57 +03:00
daavoo 64beb1578a repro: Add --allow-missing. 
Point 2 in https://github.com/iterative/dvc/pull/9375#issuecomment-1531683194
 2023-05-16 22:43:24 +02:00
daavoo 1e538ed87b repro: Move pull logic to inside Stage. 2023-05-16 16:11:43 +02:00
David de la Iglesia Castro 195e67c0c1 repro: pull all missing files (#9395) 
* reproduce: Update `pull` to also pull `run cache`.

* reproduce: Update `pull` to also pull changed stages.
 2023-05-10 00:29:44 +03:00
David de la Iglesia Castro 981baa8c30 studio: Send message on start event. 
- Bump `dvc-studio-client>=0.9.0,<1`

Part of #8870
 2023-05-08 20:08:47 +02:00
David de la Iglesia Castro aa517bc90e exp save: Support --message. 
Custom commit message to use when committing the experiment.

Part of #8870
 2023-05-08 20:08:47 +02:00
David de la Iglesia Castro c74dc20b97 exp run: Support --message. 
Custom commit message to use when committing the experiment.

Part of #8870
 2023-05-08 20:08:47 +02:00
David de la Iglesia Castro 1ba3944944 api: exp_show: Postprocess empty strings as None. 
Makes it easier to manipulate when converting to pandas DataFrame.
 2023-05-08 17:10:27 +02:00
David de la Iglesia Castro 8620b6e5af exp: Update --rev to support append action. 
Closes #9390
 2023-05-08 14:15:07 +02:00
David de la Iglesia Castro 0f1326e547 api: exp_show: Cleanup flags. 
Per https://github.com/iterative/dvc/pull/9356#discussion_r1180782320
 2023-05-01 19:29:47 +02:00
David de la Iglesia Castro 2a04263f02 api: Add scm module with all_{X} methods for listing revs. 2023-04-28 12:33:03 +02:00
David de la Iglesia Castro 348446050e api: Add exp_show. 
Closes #8775
Closes #7167
 2023-04-28 12:33:03 +02:00
David de la Iglesia Castro 58f10a4661 gc: Add ui hint for not_in_remote. 2023-04-27 10:30:39 +02:00
David de la Iglesia Castro 64e787fa02 gc: Handle remote field in --cloud and --not-in-remote. 2023-04-27 10:30:39 +02:00
David de la Iglesia Castro c7e2ec9707 gc: Add --not-in-remote arg. 
Closes #2036
 2023-04-27 10:30:39 +02:00
David de la Iglesia Castro baeec8ec2d Update pyproject.toml 2023-04-26 10:38:08 +02:00
David de la Iglesia Castro b6d8c07d4d studio: Ensure studio_repo_url works with exp run --queue. 
Closes #9349
 2023-04-21 21:04:48 +02:00
David de la Iglesia Castro 3312a90949 exp run: Add --copy-paths arg. 
List of paths to copy inside the temp directory. Only used if `--temp` or `--queue` is specified.

Closes #5800
 2023-04-05 23:17:07 +02:00
David de la Iglesia Castro 35648134f7 exp run: git add param files that will be modified. 
Only if they are currently untracked and `--temp` or `--queue` is passed.

Closes #7842
 2023-04-05 10:28:18 +02:00
David de la Iglesia Castro 3a2259821b exp save: Include LOCK_FILE 2023-04-03 10:33:52 +02:00
David de la Iglesia Castro 24e10e0278 exp save: Track top level files 2023-03-31 19:23:40 +02:00
David de la Iglesia Castro 7c8e6e3156 exp run: Track top level files. 2023-03-31 19:23:40 +02:00
David de la Iglesia Castro d2812fa48f studio: Don't fail on empty params. 2023-03-31 19:23:40 +02:00
David de la Iglesia Castro 1d8b2e5fa6 exp run: Propagate .dvc/config.local in TempDirExecutor. 
Closes #7254
 2023-03-31 09:02:22 +02:00
David de la Iglesia Castro 894f530d22 deps: bump dvc-studio-client>=0.6.1,<1 2023-03-30 11:23:07 +02:00
David de la Iglesia Castro fe6d543c1f tests: Add UnresolvedExpNamesError check in test_remove_remote 2023-03-29 12:56:59 +02:00
David de la Iglesia CastroandDave Berenbaum e9fb0b92aa exp branch: Make branch arg optional. 
Default to `f"{ref_info.name}-branch"` where ref_info is inferred from the `exp_rev` argument.

Co-authored-by: Dave Berenbaum <[email protected]>
 2023-03-24 19:11:06 +01:00
David de la Iglesia Castro 6b6dc01ac4 exp save: Reintroduce implementation based on executor/queue. 
Closes #9058 .

Ensures behavior of `_stash_exp` in `exp run` is matched in `exp save` without duplicating logic.

The simpler implementation ended up being worse for maintenance when wanting to make `exp run` and `exp save` behavior in sync.
 2023-03-24 10:43:24 +01:00
David de la Iglesia Castro 9b329d01bf api: Add exp_save. 2023-03-23 10:33:55 +01:00
David de la Iglesia Castro 121d56b739 hydra: resolve OmegaConf interpolations. (#9212) 
Closes #9196
 2023-03-20 21:38:22 +05:45
David de la Iglesia Castro 2d820cccad deps: bump dvc-render>=0.3.1<0.4.0. 
For https://github.com/iterative/dvc-render/pull/124
 2023-03-16 21:36:02 +01:00
David de la Iglesia Castro e73cc9b40a exp: Include STUDIO_REPO_URL in run_env. 
Closes #8951.
 2023-03-14 13:43:03 +01:00
David de la Iglesia Castro f1ea64c4ef studio: Send metrics on done event. (#9027) 2023-03-07 19:47:22 +01:00
David de la Iglesia Castro dde456c923 plots: Fix match_defs_renderers when plots are defined in nt format. 
`plots.show` returned `definitions` in native `nt` format but sources in `posixpath` format so the matching was failing.

Closes #8689

- Add `test_show_plots_defined_with_native_os_path`.
 2023-03-06 19:16:29 +01:00
David de la Iglesia Castro cb9d3529af plots: Remove --from-config 2023-03-06 19:16:29 +01:00
David de la Iglesia Castro 876c10fcf5 plots: Remove unnecessary OrderedDict usage in load_sv. 2023-03-06 19:16:29 +01:00
David de la Iglesia Castro e4d514c990 scm: Remove unused GitMergeError. 
Not used after https://github.com/iterative/dvc/pull/9096
 2023-03-03 18:09:57 +01:00
David de la Iglesia Castro baeecd1972 deps: Bump scmrepo>=0.1.14 2023-03-03 15:42:41 +01:00
David de la Iglesia Castro dbd098506c deps: Drop dvclive. 
See https://github.com/iterative/dvclive/pull/481
 2023-03-03 14:45:48 +01:00
David de la Iglesia Castro c6015a05a3 exp save: Don't relink during commit 
Same change as in https://github.com/iterative/dvc/pull/9076 but for `exp save`
 2023-02-28 22:29:57 +02:00
David de la Iglesia Castro 9f0c4350bd deps: Bump scmrepo>=0.1.13. 
Closes #9080
 2023-02-28 17:46:09 +09:00
David de la Iglesia Castro 184cc35cf0 pull: Add allow_missing flag. 
Allows `pull` to succeed even if some files or directories are missing.

Closes #4746
 2023-02-24 09:11:50 +02:00
David de la Iglesia Castro 3043b6588f exp save: Use pathspec to handle include_untracked. 
Closes #9061
 2023-02-21 09:43:25 +01:00
David de la Iglesia Castro b7b7eff4a3 tests: Fix test_deleted. 
Closes #6297
#9017 actually fixed this but the test was incorrect (something wrong with caplog).
 2023-02-16 11:43:09 +01:00
David de la Iglesia Castro 320771551b exp: Preserve staged changes after experiments. 
- Detach head after stashing workspace.
- Use reinstate_index.
 2023-02-14 16:59:10 +01:00
David de la Iglesia Castro 1676d1f344 Add test_stash_exp. 2023-02-14 16:59:10 +01:00
David de la Iglesia Castro c8ae2895f0 deps: bump scmrepo==0.1.10 
https://github.com/iterative/scmrepo/releases/tag/0.1.10
 2023-02-14 16:59:10 +01:00
David de la Iglesia Castro e910dac0dc Update pyproject.toml 2023-02-10 10:59:32 +01:00
David de la Iglesia Castro 17a71d6011 plots: Rename template_path to html_template. 2023-02-10 10:59:32 +01:00
David de la Iglesia Castroskshetrypre-commit-ci[bot]
 		d84196db62 studio: Move post_live_metrics from queue to executor (#8955) 
* studio: Move `post_live_metrics` from `queue/workspace` to `executor/base`.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update dvc/repo/experiments/utils.py

Co-authored-by: skshetry <[email protected]>

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: skshetry <[email protected]>
 2023-02-06 13:31:10 +00:00
David de la Iglesia Castro 12dd746165 output: Pass version_aware=True to fs if files is present. 
This will be ignored by filesystems that don't support `version_aware`.

Closes #8853
 2023-01-26 10:48:59 +01:00
David de la Iglesia Castro 80633f4fad stage.save: Remove merge_versioned option. 
If we want `add`, `commit` and `move` to work for cloud versioning, option must be always True so it is the same as not having it.

Closes #8828
(We were not passing `merge_versioned=True` in `dvc commit`)
 2023-01-25 10:35:37 +01:00
David de la Iglesia Castro ea2e5dc3e5 output: Don't serialize hash_info or meta in cloud versioning. 
Closes #8357
 2023-01-24 18:40:39 +01:00
David de la Iglesia CastroandPeter Rowlands dd2d2dce19 param: Fix status check for Tuples. 
Closes #8803

Update dvc/dependency/param.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
 2023-01-19 08:19:16 +01:00
David de la Iglesia Castro 5f558d003a deps: bump dvc-data to 0.33.0 
- Update `test_gc`
 2023-01-18 15:17:07 +02:00
David de la Iglesia Castro 3676f35b7c output.remove: Pass explicit recursive flag. 
Per changes in https://github.com/iterative/dvc-objects/pull/176
 2023-01-18 15:17:07 +02:00
David de la Iglesia Castro 9af6fcad4a exp run: Fix --dry option. 
Closes #8831
 2023-01-17 13:32:20 +01:00
David de la Iglesia Castro 8b075eb754 stage: Skip run-cache if any out has cache: false. 
Closes https://github.com/iterative/dvc/issues/4428
 2023-01-12 19:10:01 +01:00
David de la Iglesia Castro f507e4d9c0 config: Reraise ConfigError from ConfigObjError. 
Closes #8801
 2023-01-12 18:11:54 +01:00
David de la Iglesia Castro 1d5de9c5ba repro: Disable run-cache if --no-commit. 
As described in https://github.com/iterative/dvc/issues/4428#issuecomment-778263005.

Add `run_cache` argument to `Stage.save`. Set it to `False` if `no_commit`.
 2023-01-09 12:38:57 +01:00
David de la Iglesia Castro 68b24307f9 post_live_metrics: Pass params on start event. 2022-12-28 11:43:37 +01:00
David de la Iglesia Castro b13184fcec tests: Fix flaky test_show_only_changed. 
Some of the random exp names could start with `bar`.

https://github.com/iterative/dvc/actions/runs/3757008396/jobs/6383731723#step:5:79
 2022-12-22 13:01:46 +01:00
David de la Iglesia Castro d7fd68a1ae Add start and done calls with post_live_metrics. 2022-12-22 11:33:59 +01:00
David de la Iglesia Castro dc3d9c9677 bump DVCLive lower bound 2022-12-14 20:45:57 +01:00
David de la Iglesia CastroandDave Berenbaum c806275167 Bump dvclive lower bound (#8683) 
* Bump dvclive lower bound

* Update pyproject.toml

Co-authored-by: Dave Berenbaum <[email protected]>

Co-authored-by: Dave Berenbaum <[email protected]>
 2022-12-14 14:13:20 +00:00
David de la Iglesia Castro bb7512687c exp: Generate a human-readable name beforehand. 
- Add `get_random_exp_name` to utils.
Generates a name (`f"{adjective}-{noun}"`) by randomly chosing pairs of adjetive, noun.
Recieves `scm` and `baseline_rev` and retries the random choice until a non-existing ref is found.

Closes https://github.com/iterative/dvc/issues/8650
 2022-12-09 10:32:38 +01:00
David de la Iglesia Castro a036eaf817 exp: save: Add include_untracked. 
Allow to include a list of potentially untracked files.
Covers the DVCLive use case where in the first run the generated files won't be tracked in Git yet.
 2022-11-28 15:49:34 +01:00
David de la Iglesia Castro 665185b9e7 exp: save: Handle staged changes. 
Staged changes were causing a merge error.
Warn and unstage instead of erroring, to match exp run behavior.
 2022-11-28 15:49:34 +01:00
David de la Iglesia Castro 43e281521e exp save: cleanup. 
Remove metrics.

Move tests to separate file.

Fix `experiments.get_exact_name` usage.

Remove unused checkpoint logic.
 2022-11-28 15:49:34 +01:00
David de la Iglesia Castro 8124396218 exp: Expose baseline and name via run_env. 
Required by DVCLive for the experiments onboarding and live metrics features.

Related (not sure if closes): https://github.com/iterative/dvc/issues/6668
 2022-11-28 13:13:04 +01:00
David de la Iglesia CastroPeter Rowlandspre-commit-ci[bot]
 		b02855fb93 ui: Fix WSL check in open_browser (#8604) 
* ui: Fix WSL check in `open_browser`

* Update dvc/ui/__init__.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2022-11-23 09:57:57 +00:00
David de la Iglesia Castro 705bc6ec82 Added test_infer_x_label 2022-11-21 21:05:12 +01:00
David de la Iglesia CastroandDave Berenbaum b0a92414fa hydra: Use OmegaConf.to_yaml for dumping .yaml output. (#8587) 
* hydra: Use `OmegaConf.to_yaml` for dumping `.yaml` output.

Combining `OmegaConf.to_object` with our yaml DUMPER (ruaml.yaml) was not correctly handling strings like 'no'.

`OmegaConf.to_yaml` uses a custom string representer that correctly handles those cases.

Fixes #8583

* Update dvc/utils/hydra.py

Co-authored-by: Dave Berenbaum <[email protected]>

Co-authored-by: Dave Berenbaum <[email protected]>
 2022-11-19 12:37:18 +00:00
David de la Iglesia Castro a33bbc97d5 parsing: Escape str interpolation in dict unpacking. 
Use subprocess.list2cmdline (windows) / shlex.quote

Closes #8203
Closes #8370
 2022-11-18 20:00:16 +01:00
David de la Iglesia Castro e4ea463c73 hydra: Raise error when name and sweeps. 2022-11-14 21:34:55 +01:00
David de la Iglesia Castro 1de983f0de Remove live section support. 2022-11-07 12:45:32 +01:00
David de la Iglesia Castro eb20a8e049 deps: Bump dvclive to >= 1.0 2022-11-07 12:45:32 +01:00
David de la Iglesia Castro 78e25747a4 exp init: Update DVCLive paths. 
Closes #8531
 2022-11-07 12:45:32 +01:00
David de la Iglesia Castro 08a643e005 hydra: Raise lazy DvcException for Python >= 3.11 2022-11-03 12:14:18 +01:00
David de la Iglesia Castro 61f8478885 tests: Remove test_ssh 
Moved to dvc-ssh repo in https://github.com/iterative/dvc-ssh/pull/13
 2022-10-19 12:35:11 +03:00
David de la Iglesia Castro 3bbf8fa7d2 config: ssh: Add passphrase and ask_passphrase options. 
Bump `dvc-ssh`.

Fixes #8137
 2022-10-14 12:11:31 +03:00
David de la Iglesia Castro c12deaa62c exp show: Preserve full branch and tag names. 
Closes #7933
 2022-10-11 19:55:36 +09:00
David de la Iglesia Castro cf651c52a1 hydra: Fix append and remove sweeps. 
Closes #8376
 2022-10-04 19:05:12 +02:00
David de la Iglesia Castro 122a68398e hydra: Fix sweeps on Defaults List. 
Defaults List overrides (`defaults/foo=1,2`) were not working because of the inclusion of the `/` character.
Fixed it by updating how we construct the `merged_overrides` and added a regression test.
 2022-09-21 15:48:19 +02:00
David de la Iglesia Castro 3b532b7d59 exp run: Support hydra basic sweeper. 
Requires `--queue` to be used alongside `--set-param`.

Doesn't strictly require `hydra.config.enabled`.

Add `get_hydra_sweeps` to `dvc/utils/hydra.py`.  Uses hydra `BasicSweeper` to generate the different sweeps.

`glob` syntax for sweeps is not supported in this P.R.

Closes #8083

---

Using setup from https://hydra.cc/docs/tutorials/basic/your_first_app/composition/ :

```console
$ dvc exp run -S "db=mysql,postgresql" -S "schema=warehouse,school" --queue
Queueing with '{'params.yaml': ['db=mysql', 'schema=warehouse']}'.
Queued experiment '5ab98b8' for future execution.
Queueing with '{'params.yaml': ['db=mysql', 'schema=school']}'.
Queued experiment '57c2fb6' for future execution.
Queueing with '{'params.yaml': ['db=postgresql', 'schema=warehouse']}'.
Queued experiment 'b9d6391' for future execution.
Queueing with '{'params.yaml': ['db=postgresql', 'schema=school']}'.
Queued experiment '145cd55' for future execution.
```
 2022-09-13 13:43:30 +02:00
David de la Iglesia Castro 91f12395f1 exp run: Support composing and dumping Hydra config. 
The feature will be used depending on whether `config.hydra.enabled` is True or False.

Uses `hydra.initialize_config_dir` and `hydra.compose` (from https://hydra.cc/docs/advanced/compose_api/) to build the config and dump it to `params.yaml`. The content of the output file will be overwritten.

`config.hydra.config_dir` and `config.hydra.config_name` can be used to customize the values passed to `hydra.initialize_config_dir` and `hydra.compose`.

Can be combined with `--set-param` overrides.

Closes #8082
 2022-08-31 17:24:09 +02:00
David de la Iglesia Castro daee52e176 info: skip missing subprojects 2022-08-30 20:15:41 +03:00
David de la Iglesia Castro ab780879d1 info: Include subprojects. 2022-08-30 09:22:39 +02:00
David de la Iglesia Castro 9e2ebe9fcc resolve_rev: raise DvcException on empty repo. 2022-08-29 01:50:14 +03:00