Commit Graph
1611 Commits
Author SHA1 Message Date
David de la Iglesia Castro dde456c923 plots: Fix match_defs_renderers when plots are defined in nt format.
`plots.show` returned `definitions` in native `nt` format but sources in `posixpath` format so the matching was failing.

Closes #8689

- Add `test_show_plots_defined_with_native_os_path`.
2023-03-06 19:16:29 +01:00
David de la Iglesia Castro cb9d3529af plots: Remove --from-config 2023-03-06 19:16:29 +01:00
skshetry 93eea0cd42 deps: bump scmrepo to 0.1.15; close subrepo.dvc on tests teardown (#9122) 2023-03-06 20:21:42 +05:45
skshetry ddfa607437 tests: don't leak files/descriptors when tests finish (#9119)
close open files/descriptors when tests finish
2023-03-06 18:06:37 +05:45
Peter Rowlands 2cf3a12e46 tests: unmark exp celery tests as xfail 2023-03-02 15:44:34 +09:00
Peter Rowlands 78fb5275e5 exp: add dvc exp clean command 2023-03-02 12:35:54 +09:00
Ruslan Kuprieiev 59a4c4f67c remote: set hash_name as md5
Fixes #7977
2023-03-01 13:26:24 +02:00
David de la Iglesia Castro 184cc35cf0 pull: Add allow_missing flag.
Allows `pull` to succeed even if some files or directories are missing.

Closes #4746
2023-02-24 09:11:50 +02:00
Peter Rowlands ad5b91e51a exp show: cache collected git revs 2023-02-24 14:18:58 +09:00
Ruslan Kuprieiev 37761a2627 data: status: get rid of --not-in-remote 2023-02-21 12:32:40 +02:00
David de la Iglesia Castro 3043b6588f exp save: Use pathspec to handle include_untracked.
Closes #9061
2023-02-21 09:43:25 +01:00
47208533a2 Add metrics_show to dvc.api (#9042)
* refactor: add metrics_show, rename module to show

* tests: add functional tests for metrics_show

* api.show: return empty dict for no params/metrics

* test_show: Return relpaths in `metrics_repo`.

---------

Co-authored-by: codamuse <[email protected]>
Co-authored-by: David de la Iglesia Castro <[email protected]>
2023-02-20 21:53:38 +00:00
Ruslan Kuprieiev c34cbf63af data: status: introduce --not-in-remote
Also supports cloud versioning.

Fixes #8761
2023-02-20 12:25:17 +02:00
Peter Rowlands 648edd71eb exp: move apply tests into their own module 2023-02-20 14:36:38 +09:00
Peter Rowlands 0c95cde712 exp: remove unused ApplyConflictError 2023-02-20 14:36:38 +09:00
skshetry 1de37556d5 bump ruff to v0.0.247 (#9050)
* bump ruff to v0.0.247

* ruff: enable pep8-naming check

rename some camelCased APIs, especially in logger
rename G variables to graph
ignore some variable namings and help messages in dvc.commands
2023-02-18 16:50:35 +05:45
David de la Iglesia Castro b7b7eff4a3 tests: Fix test_deleted.
Closes #6297
#9017 actually fixed this but the test was incorrect (something wrong with caplog).
2023-02-16 11:43:09 +01:00
Ruslan Kuprieiev 49b5157abe diff: use index 2023-02-15 06:40:02 +02:00
David de la Iglesia Castro 320771551b exp: Preserve staged changes after experiments.
- Detach head after stashing workspace.
- Use reinstate_index.
2023-02-14 16:59:10 +01:00
David de la Iglesia Castro 1676d1f344 Add test_stash_exp. 2023-02-14 16:59:10 +01:00
Karan Jakhar 73497dae9e Handle filenames which has ":" (#9000)
* Fixes #8773 issue

* added test for #8773 issue

* skip test for windows
2023-02-14 11:21:32 +00:00
Ruslan Kuprieiev e73d662686 repo: introduce data_index
Extracting from `index.data` as a "root" data index that contains indexes
for different revisions/workspaces.

This is a setup for putting cloud versioning remote indexes and also replacing
old state.
2023-02-13 06:31:44 +02:00
Ruslan Kuprieiev 58295ca0ab repo: rename odb -> cache
We used to have it named `cache` before, but these days it is clear that
you can have three odbs: just odb (e.g. in-memory), cache (typically local)
or remote (typically cloud). Fixing naming here before proceeding with
introducing `out.odb` and `out.cache`.

In preparation for migration to https://github.com/iterative/dvc-data/pull/304
2023-02-08 23:05:45 +02:00
Hagai Kariti 6c8c26f741 Support setting a exp name prefix in hydra sweeps 2023-02-07 18:49:23 +01:00
d84196db62 studio: Move post_live_metrics from queue to executor (#8955)
* studio: Move `post_live_metrics` from `queue/workspace` to `executor/base`.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update dvc/repo/experiments/utils.py

Co-authored-by: skshetry <[email protected]>

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: skshetry <[email protected]>
2023-02-06 13:31:10 +00:00
Ruslan Kuprieiev 8e1b75feb5 index: build_data_index: handle empty dirs (#8959)
Fixes https://github.com/iterative/dvc/issues/8958
2023-02-06 01:44:15 +02:00
skshetry a86470ccbc update black and change line-length to 88 (#8953) 2023-02-04 05:28:07 +02:00
Ruslan Kuprieiev 24023c6977 index: introduce data_keys (#8938)
`data_keys` are similar to `outs`, but they provide more granularity, so we
can use them in `build_data_index` to rebuild index views with filtered
outputs. E.g. when running `dvc diff dir/file`, workspace index should only
have `dir/file` in it.

This also could be used for standalone (non-output) data in the future.

Pre-requisite for #8930
2023-02-02 05:50:16 +02:00
Ruslan Kuprieiev f661cee8f6 stage: use repo.fs directly without copying (#8929)
Otherwise when using brancher we end up in a strange situation where even
though we are collecting targets from `gitfs`, we are still using `localfs`
as `self.fs`, which make operations like `self.fs.abspath` in
`collect_granular` return invalid paths.
2023-02-01 05:21:53 +02:00
Ruslan Kuprieiev e07c85a993 index: collect_targets: introduce onerror (#8925) 2023-02-01 03:49:27 +02:00
Ruslan Kuprieiev 58d9e379fe tests: gc: use timedelta again (#8926)
Leftover from #8912
2023-02-01 03:14:22 +02:00
Ruslan Kuprieiev 10184e9370 tests: gc: use timedelta (#8912)
Bug from https://github.com/iterative/dvc/pull/8880
2023-01-31 05:18:09 +02:00
98bcfad8e7 build(deps-dev): bump scmrepo from 0.1.6 to 0.1.7 (#8821)
* build(deps-dev): bump scmrepo from 0.1.6 to 0.1.7

Bumps [scmrepo](https://github.com/iterative/scmrepo) from 0.1.6 to 0.1.7.
- [Release notes](https://github.com/iterative/scmrepo/releases)
- [Commits](https://github.com/iterative/scmrepo/compare/0.1.6...0.1.7)

---
updated-dependencies:
- dependency-name: scmrepo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* mypy: remove scmrepo from ignored modules

* Fix mypy issues

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: skshetry <[email protected]>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-01-28 14:26:00 +05:45
skshetry 7ab4c59d70 test: run pytest-style checks (#8901) 2023-01-28 14:10:53 +05:45
skshetry a137b2a6bc update ruff and enable flake8-logging-format and flake8-type-checking (#8894)
* update ruff and enable flake8-logging-format and flake8-type-checking check

* Update dvc/external_repo.py

* Update dvc/fs/dvc.py

* Update dvc/repo/experiments/__init__.py

* Update dvc/repo/experiments/queue/base.py

* Update pyproject.toml
2023-01-27 19:14:19 +05:45
Daniele Trifirò d294945499 gc: fix usage with --date 2023-01-27 12:52:46 +01:00
5165fb67a1 [pre-commit.ci] pre-commit autoupdate (#8869)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/charliermarsh/ruff-pre-commit: v0.0.223 → v0.0.230](https://github.com/charliermarsh/ruff-pre-commit/compare/v0.0.223...v0.0.230)

* Update .pre-commit-config.yaml

* fix lint issues

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: skshetry <[email protected]>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-01-26 14:24:09 +05:45
Ruslan Kuprieiev b46cfbda5d deps: bump dvc-data to 0.36.0 (#8885) 2023-01-26 00:22:12 +02:00
skshetry 3427e115eb data status: check if change.old.hash_info is None or not (#8878)
data:status: check if change.old.hash_info is None or not
2023-01-25 14:24:38 +02:00
David de la Iglesia Castro 80633f4fad stage.save: Remove merge_versioned option.
If we want `add`, `commit` and `move` to work for cloud versioning, option must be always True so it is the same as not having it.

Closes #8828
(We were not passing `merge_versioned=True` in `dvc commit`)
2023-01-25 10:35:37 +01:00
David de la Iglesia Castro ea2e5dc3e5 output: Don't serialize hash_info or meta in cloud versioning.
Closes #8357
2023-01-24 18:40:39 +01:00
Peter Rowlands 38e9b2ba95 deps: bump dvc-data to 0.35.1 2023-01-21 03:44:02 +09:00
David de la Iglesia Castro 5f558d003a deps: bump dvc-data to 0.33.0
- Update `test_gc`
2023-01-18 15:17:07 +02:00
David de la Iglesia Castro 9af6fcad4a exp run: Fix --dry option.
Closes #8831
2023-01-17 13:32:20 +01:00
skshetry c47c2123fb cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc (#8822)
* cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc.

* remove pylint direcitves

* Apply suggestions from code review

* Update pyproject.toml
2023-01-17 08:35:19 +05:45
Peter Rowlands 977a1f9f43 tests: update for dvc-objects batched transfer 2023-01-16 15:58:21 +09:00
skshetry b0052aa666 pre-commit update: flake8/ruff/pyupgrade/black/isort (#8813) 2023-01-15 14:56:54 +05:45
skshetry da27174e02 dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups, etc. (#8812)
dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups etc
2023-01-15 12:45:10 +05:45
karajan1001 fc1a192c90 Bump pathspec to 0.10.3
fix: #8217
1. Replace group name to avoid conflict.
2. Disable failed tests.
3. Bump pathspec to 0.10.3
2023-01-13 22:47:36 +08:00
dependabot[bot]andSaugat Pachhai 178244f666 build(deps-dev): bump dvc-task from 0.1.9 to 0.1.10 (#8807)
* build(deps-dev): bump dvc-task from 0.1.9 to 0.1.10

Bumps [dvc-task](https://github.com/iterative/dvc-task) from 0.1.9 to 0.1.10.
- [Release notes](https://github.com/iterative/dvc-task/releases)
- [Commits](https://github.com/iterative/dvc-task/compare/0.1.9...0.1.10)

---
updated-dependencies:
- dependency-name: dvc-task
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* remove xfail on json.JSONDecodeError/PermissionError

JSONDecodeError should be fixed by the new release. Next we need to investigate
PermissionError.

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-01-13 17:25:36 +05:45