Commit Graph
2982 Commits
Author SHA1 Message Date
Dave Berenbaum db70d1cc5c analytics: differentiate ci users (#8838) 2023-01-26 21:50:14 +05:45
David de la Iglesia Castro 12dd746165 output: Pass version_aware=True to fs if files is present.
This will be ignored by filesystems that don't support `version_aware`.

Closes #8853
2023-01-26 10:48:59 +01:00
5165fb67a1 [pre-commit.ci] pre-commit autoupdate (#8869)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/charliermarsh/ruff-pre-commit: v0.0.223 → v0.0.230](https://github.com/charliermarsh/ruff-pre-commit/compare/v0.0.223...v0.0.230)

* Update .pre-commit-config.yaml

* fix lint issues

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: skshetry <[email protected]>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-01-26 14:24:09 +05:45
Ruslan Kuprieiev b46cfbda5d deps: bump dvc-data to 0.36.0 (#8885) 2023-01-26 00:22:12 +02:00
Dave Berenbaum a8d193bf2a queue: update kill ui messages (#8798)
* queue: update kill ui messages

* queue kill: update ui, kill all valid tasks

* queue kill: update wording of ui
2023-01-25 20:08:27 +05:45
skshetry 3427e115eb data status: check if change.old.hash_info is None or not (#8878)
data:status: check if change.old.hash_info is None or not
2023-01-25 14:24:38 +02:00
David de la Iglesia Castro 80633f4fad stage.save: Remove merge_versioned option.
If we want `add`, `commit` and `move` to work for cloud versioning, option must be always True so it is the same as not having it.

Closes #8828
(We were not passing `merge_versioned=True` in `dvc commit`)
2023-01-25 10:35:37 +01:00
David de la Iglesia Castro ea2e5dc3e5 output: Don't serialize hash_info or meta in cloud versioning.
Closes #8357
2023-01-24 18:40:39 +01:00
Peter Rowlands 38e9b2ba95 deps: bump dvc-data to 0.35.1 2023-01-21 03:44:02 +09:00
David de la Iglesia CastroandPeter Rowlands dd2d2dce19 param: Fix status check for Tuples.
Closes #8803

Update dvc/dependency/param.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2023-01-19 08:19:16 +01:00
David de la Iglesia Castro 5f558d003a deps: bump dvc-data to 0.33.0
- Update `test_gc`
2023-01-18 15:17:07 +02:00
David de la Iglesia Castro 9af6fcad4a exp run: Fix --dry option.
Closes #8831
2023-01-17 13:32:20 +01:00
skshetry c47c2123fb cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc (#8822)
* cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc.

* remove pylint direcitves

* Apply suggestions from code review

* Update pyproject.toml
2023-01-17 08:35:19 +05:45
Peter Rowlands 977a1f9f43 tests: update for dvc-objects batched transfer 2023-01-16 15:58:21 +09:00
Peter Rowlands e48d0d1d9c tests: remove duplicated dvc-objects/dvc-data tests 2023-01-16 15:58:21 +09:00
skshetry b0052aa666 pre-commit update: flake8/ruff/pyupgrade/black/isort (#8813) 2023-01-15 14:56:54 +05:45
skshetry da27174e02 dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups, etc. (#8812)
dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups etc
2023-01-15 12:45:10 +05:45
karajan1001 fc1a192c90 Bump pathspec to 0.10.3
fix: #8217
1. Replace group name to avoid conflict.
2. Disable failed tests.
3. Bump pathspec to 0.10.3
2023-01-13 22:47:36 +08:00
dependabot[bot]andSaugat Pachhai 178244f666 build(deps-dev): bump dvc-task from 0.1.9 to 0.1.10 (#8807)
* build(deps-dev): bump dvc-task from 0.1.9 to 0.1.10

Bumps [dvc-task](https://github.com/iterative/dvc-task) from 0.1.9 to 0.1.10.
- [Release notes](https://github.com/iterative/dvc-task/releases)
- [Commits](https://github.com/iterative/dvc-task/compare/0.1.9...0.1.10)

---
updated-dependencies:
- dependency-name: dvc-task
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* remove xfail on json.JSONDecodeError/PermissionError

JSONDecodeError should be fixed by the new release. Next we need to investigate
PermissionError.

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-01-13 17:25:36 +05:45
David de la Iglesia Castro 8b075eb754 stage: Skip run-cache if any out has cache: false.
Closes https://github.com/iterative/dvc/issues/4428
2023-01-12 19:10:01 +01:00
David de la Iglesia Castro f507e4d9c0 config: Reraise ConfigError from ConfigObjError.
Closes #8801
2023-01-12 18:11:54 +01:00
Saugat Pachhai (सौगात) 30ec1cf5e9 index/stage: move repo collection logic to index 2023-01-11 19:38:43 +02:00
skshetry a814f04ee0 tests: xfail test_apply_failed on JSONDecodeError/PermissionError (#8796) 2023-01-11 18:04:23 +05:45
skshetry 90a27179eb ruff: ignore complex functions and remove dvc.utils.fs.makedirs (#8795)
We may want to prevent any new functions from getting complex.
2023-01-11 17:58:51 +05:45
skshetry 175c578579 ruff: ban relative parent/grandparent imports (#8794) 2023-01-11 16:48:46 +05:45
skshetry 5441964fdb add ruff (#8793)
* add ruff linter

* enable few more checks
2023-01-11 15:20:38 +05:45
Ivan Shcheklein d4bfe38a54 perf: remove fs exists check in plots, parallel data collect 2023-01-10 20:27:02 -08:00
skshetry d87cdba6d8 gha: remove env vars SHELL 2023-01-10 10:19:29 +05:45
dberenbaum 0b192af6f4 params: fix diff bt active branch and workspace 2023-01-09 16:57:18 +02:00
dberenbaum c6b602a167 metrics: fix diff bt active branch and workspace 2023-01-09 16:57:18 +02:00
David de la Iglesia Castro 1d5de9c5ba repro: Disable run-cache if --no-commit.
As described in https://github.com/iterative/dvc/issues/4428#issuecomment-778263005.

Add `run_cache` argument to `Stage.save`. Set it to `False` if `no_commit`.
2023-01-09 12:38:57 +01:00
Ruslan Kuprieiev 684fc32e36 dvcfs: ls: don't call info()
Cuts down `dvc ls . data/mnist/dataset --recursive` for me from ~37 to ~25
seconds.

Note that this also changes behaviour for ls with a broken symlink, as we
were not able to see it before because of a failing `info()` call, but now
we can follow a normal UNIX ls behaviour more closely.
2023-01-08 00:37:40 +02:00
Saugat Pachhai (सौगात) 3dfdd577e7 tests: xfail experiments-related flakey tests 2023-01-06 18:59:53 +05:45
Daniele Trifirò b62f17ffaf tests: move tests/unit/remote/test_http.py to dvc-http repo 2023-01-05 19:54:14 +01:00
dberenbaum 6ac7dfc008 exp: fix tests and remove reset checks 2023-01-04 17:29:12 +01:00
dberenbaum b0372918ea exp: stop checking for unchanged experiments 2023-01-04 17:29:12 +01:00
Daniele Trifirò 4b0fa4d559 update: prevent re-download on every update 2023-01-03 18:52:48 +01:00
Saugat Pachhai (सौगात) a24c389674 internal: hoist resolver/top-level metrics/plots/params data into file level 2023-01-03 17:46:06 +05:45
karajan1001 f386f7c082 exp apply can accept exp names and failed exps as target
fix: #8182

1. `exp apply` can apply to failed experiments.
2. `exp apply` can accept exp names as target.
3. Add a new functional test for it.
2022-12-30 17:40:18 +08:00
karajan1001 eadd0da394 Refactor get_ref_and_entry_by_names
1. Move get_ref_and_entry_by_names from `exp remove` to `celery` for
   other commands usage.
2022-12-30 17:40:18 +08:00
karajan1001 06673dd212 Use SIGINT as the default signal in queue kill
fix: #8624
1. Add a new flag `--force` for `queue kill`
2. Make `SIGINT` to be the default option and `SIGKILL` to be with
   `--force`
3. Add tests for `queue kill`
4. Bump dvc-task into 0.1.9
2022-12-30 16:19:39 +08:00
David de la Iglesia Castro 68b24307f9 post_live_metrics: Pass params on start event. 2022-12-28 11:43:37 +01:00
karajan1001 08ae68caf4 Make temp dir executed checkpoint experiment return result to workspace.
fix: #8612
For checkpoint experiment in some case users might want to give it a early
stopping to reduce variance.  But currently if we interrupt/kill the
experiment it will be marked as failed, and all of the completed
checkpoints will be removed as we cleanup all the running directly after
the process failed.

1. We raise CheckpointKilledError intead of StageCmdFailed error if at
   least one checkpoint had been commited.
2. Temp_dir executor will continue collecting data if the Checkpoint
   stage was interrupted.
3. Raise warnings if a checkpoint stage was incomplete and the other
   stages were not forwarded.
4. Add a new functional test for this
2022-12-26 18:25:43 +05:45
karajan1001 4400fb4090 queue stop --kill can kill false running status tasks
fix: #8702

1. replace simple `proc.kill` in `queue stop --kill` with the same logic as in
   `queue kill` to make it possible to set false status `running` tasks
    as faileed.
2. Modified the tests.
2022-12-26 17:16:46 +09:00
Fabian Zills 28848c5407 exp run --run-all: re-enable --jobs concurrency (#8608)
* re-enable --jobs concurrency

* use `start_workers` instead of `spawn_worker`

* make `spawn_worker` a private method.

* update tests: make `spawn_worker` a private method.
2022-12-26 16:49:46 +09:00
David de la Iglesia Castro b13184fcec tests: Fix flaky test_show_only_changed.
Some of the random exp names could start with `bar`.

https://github.com/iterative/dvc/actions/runs/3757008396/jobs/6383731723#step:5:79
2022-12-22 13:01:46 +01:00
David de la Iglesia Castro d7fd68a1ae Add start and done calls with post_live_metrics. 2022-12-22 11:33:59 +01:00
Daniele Trifirò 59d6d5ea98 exp save: simplify implementation 2022-12-22 11:01:41 +01:00
dberenbaum 6fc0a6ff98 plots: accept any number of y data sources 2022-12-20 14:32:40 +01:00
dberenbaum b4b5db4b4e plots: accept list of y vals 2022-12-20 14:32:40 +01:00