Compare commits

...
63 Commits
Author SHA1 Message Date
pre-commit-ci[bot]andSaugat Pachhai 356dfa0327 [pre-commit.ci] pre-commit autoupdate (#10981)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2026-03-31 10:49:52 +05:45
skshetry 0f4e8a23c8 fix(ls): support tabulate 0.10.0 preserve_whitespace (#11021) 2026-03-31 10:32:56 +05:45
dependabot[bot] 333edb2cbf build(deps): bump codecov/codecov-action from 5 to 6 (#11020)
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 5 to 6.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-31 09:26:06 +05:45
Ivan Shcheklein a65523a86a feat(gc): allow num with all-branches (#11013) 2026-03-18 22:21:20 +05:45
Christian W. Feldmann fb42429107 Resolve PyparsingDeprecationWarning for enablePackrat setParseAction parseString (#11016) 2026-03-18 22:02:17 +05:45
dependabot[bot] e40e378fc9 build(deps): bump actions/download-artifact from 7 to 8 (#11006)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-27 11:08:23 +05:45
dependabot[bot] b1aaeb94c0 build(deps): bump actions/upload-artifact from 6 to 7 (#11007)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-27 11:08:15 +05:45
skshetry dcf6f48fb3 bench: add version constraint for pathspec (#10986) 2026-02-01 09:15:42 +05:45
Caleb P. BurnsandSaugat Pachhai c6c680d2b2 Support pathspec v1 (#10951)
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2026-01-31 11:12:13 +05:45
skshetry 1e8962bf0c Remove boto3 from metadata copying in hook-dvc.py (#10983) 2026-01-28 19:08:15 +05:45
skshetryandpre-commit-ci[bot] 0f2361f218 fix(benchmarks): pin dulwich<1.0.0 for DVC versions before 3.44.0 (#10979)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-01-23 23:33:50 +05:45
pre-commit-ci[bot] e9c0a99083 [pre-commit.ci] pre-commit autoupdate (#10975)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-01-21 16:01:48 +05:45
Ritoban Dutta 3c71fdbb0d fix: replace deprecated parseAll with parse_all in interpolate.py (#10971) 2026-01-16 18:27:02 +05:45
pre-commit-ci[bot] 04cbd40df5 [pre-commit.ci] pre-commit autoupdate (#10967)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-01-13 22:44:02 +05:45
Pandikunta Anand Reddy 5725780790 refactor: drop stale exceptions related to dvc machine (#10963) 2026-01-08 21:54:13 +05:45
skshetry 2dc6261a66 Update dvc-data version to 3.18.2 (#10962) 2026-01-08 21:25:25 +05:45
skshetry 0f73c0a725 tests: add a regression test for #10959 (#10960) 2026-01-08 21:14:22 +05:45
Wyatt ff8752c3dc gc: log paths during dry-run garbage collection (#10937) 2026-01-07 18:16:05 +05:45
Wei Ji dbc7531f0d Typo on flatten-dict dependency (#10953) 2026-01-07 09:49:51 +05:45
skshetry d4594c749c fix(cli/dag): improve help message (#10949) 2026-01-06 14:34:06 +05:45
fhausmannandskshetry e062f545c9 Add option to collapse foreach/matrix stages into a single stage (#10945)
Co-authored-by: skshetry <[email protected]>
2026-01-06 08:38:08 +00:00
vorj 3c77443696 fix pathspec version requirement (#10948) 2026-01-06 11:46:26 +05:45
dependabot[bot] b5555b7285 build(deps-dev): update pytest requirement from <9,>=7 to >=7,<10 (#10946)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-06 11:16:49 +05:45
dependabot[bot] fbdcb6e140 build(deps): update flufl-lock requirement from <9,>=8.1.0 to >=8.1.0,<10 (#10947)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-06 11:16:27 +05:45
skshetry 7ed8d896df update test fixture to use matchers from dvc.testing (#10940) 2025-12-30 09:08:31 +05:45
skshetry 9fb761772c update test fixture to use tmp_path_factory (#10939) 2025-12-29 23:50:15 +05:45
skshetry f523105219 Remove pytest-test-utils from dependencies (#10938) 2025-12-28 23:51:56 +05:45
pre-commit-ci[bot] e1ccc2c7c5 [pre-commit.ci] pre-commit autoupdate (#10935)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-12-23 09:18:53 +05:45
a6594e0cc9 Use bulk calls for checking which entries are not in remote (#10923)
Co-authored-by: Sami Jawhar <[email protected]>
Co-authored-by: skshetry <[email protected]>
2025-12-22 07:42:36 +00:00
skshetry 796903a14b drop pytest-test-utils (#10934) 2025-12-20 21:26:00 +05:45
pre-commit-ci[bot] 62ae466af2 [pre-commit.ci] pre-commit autoupdate (#10931)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-12-16 09:08:45 +05:45
dependabot[bot] e6dba6ef91 build(deps-dev): bump mypy from 1.19.0 to 1.19.1 (#10932)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-16 09:08:33 +05:45
dependabot[bot] be4fd3b197 build(deps): bump actions/upload-artifact from 5 to 6 (#10929)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-15 11:14:50 +05:45
dependabot[bot] f05211113a build(deps): bump actions/download-artifact from 6 to 7 (#10930)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-15 11:14:03 +05:45
dependabot[bot] 9b42dc534e build(deps): bump actions/cache from 4 to 5 (#10928)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-12 13:40:29 +05:45
pre-commit-ci[bot] 01b0f3bb52 [pre-commit.ci] pre-commit autoupdate (#10924)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-12-09 09:02:02 +05:45
f342cc7db5 feat(webdav): add bearer_token_command for dynamic token acquisition (#10917)
Co-authored-by: GreenHatHG <[email protected]>
Co-authored-by: skshetry <[email protected]>
2025-12-05 11:07:22 +05:45
pre-commit-ci[bot] 0eb279ea35 [pre-commit.ci] pre-commit autoupdate (#10921)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-12-03 13:17:20 +05:45
Staszek Markowski d086bbbd3a exp: add support for --no-hydra flag (#10866) 2025-12-02 16:05:13 +05:45
Saugat Pachhai (सौगात) 046d2fd72e update copyright 2025-12-01 16:54:40 +05:45
skshetry cb96cf1487 fix name (#10920) 2025-12-01 15:52:37 +05:45
dependabot[bot] cf5168af74 build(deps-dev): bump mypy from 1.18.2 to 1.19.0 (#10919)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-01 12:02:39 +05:45
Saugat Pachhai (सौगात) dc123102cc fix tests 2025-11-27 09:00:19 +05:45
Saugat Pachhai (सौगात) 438259cc70 fix repository owner name to treeverse 2025-11-27 08:54:23 +05:45
pre-commit-ci[bot] 7b978fec5b [pre-commit.ci] pre-commit autoupdate (#10918)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-11-26 23:24:34 +05:45
dependabot[bot] 9f901e57a2 build(deps): bump actions/checkout from 5 to 6 (#10916)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-21 09:03:04 +05:45
pre-commit-ci[bot] ab5ac17925 [pre-commit.ci] pre-commit autoupdate (#10914)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-11-18 18:23:58 +05:45
pre-commit-ci[bot] 72abd013cf [pre-commit.ci] pre-commit autoupdate (#10908)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-11-11 10:21:29 +05:45
pre-commit-ci[bot] fa5c33ed24 [pre-commit.ci] pre-commit autoupdate (#10906)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-11-04 13:56:29 +05:45
pre-commit-ci[bot] 9dda30b7a0 [pre-commit.ci] pre-commit autoupdate (#10901)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-10-28 09:18:51 +05:45
skshetry 19c146e392 fix(data-status): iterate/expand index upto tracked directory in shallow mode (#10900) 2025-10-27 17:06:07 +00:00
dependabot[bot] 749c22bbcb build(deps): bump actions/upload-artifact from 4 to 5 (#10897)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-27 09:18:20 +05:45
dependabot[bot] 19a0bf48f1 build(deps): bump actions/download-artifact from 5 to 6 (#10898)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-27 09:18:08 +05:45
skshetry bda614baa8 support Python 3.14 (#10896) 2025-10-24 19:26:50 +05:45
pre-commit-ci[bot] d6fa8b67ff [pre-commit.ci] pre-commit autoupdate (#10894)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-10-21 09:30:04 +05:45
skshetry 498a4d5d3a move: fix error message when dst stage file exists (#10893) 2025-10-17 19:07:51 +05:45
pre-commit-ci[bot] b74711d5fe [pre-commit.ci] pre-commit autoupdate (#10892)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-10-14 13:05:03 +05:45
skshetry 61d5de54e5 support moving missing file keeping the metadata intact (#10890) 2025-10-13 05:34:14 +00:00
skshetry ca7454980d move: support moving output from import/import-url stages (#10889) 2025-10-12 17:28:41 +00:00
skshetry 8912ca7d2b exp: add a retry when removing temporary refs to avoid FileLocked errors (#10840) 2025-10-12 10:05:21 +05:45
dependabot[bot] f13ae75a45 build(deps): bump astral-sh/setup-uv from 6 to 7 (#10883)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-08 12:51:45 +05:45
dependabot[bot] aa090ecf8d build(deps): bump github/codeql-action from 3 to 4 (#10884)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-08 12:40:20 +05:45
pre-commit-ci[bot] 0c4a5cbbb7 [pre-commit.ci] pre-commit autoupdate (#10882)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.13.2 → v0.13.3](https://github.com/astral-sh/ruff-pre-commit/compare/v0.13.2...v0.13.3)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-10-07 09:05:01 +05:45
117 changed files with 1152 additions and 362 deletions
+4 -4
View File
@@ -737,8 +737,8 @@
"tests/func/test_config.py::test_config_gdrive_fields": 0.12703910000016094, "tests/func/test_config.py::test_config_gdrive_fields": 0.12703910000016094,
"tests/func/test_config.py::test_config_get[args0-0-False]": 0.20948869999995168, "tests/func/test_config.py::test_config_get[args0-0-False]": 0.20948869999995168,
"tests/func/test_config.py::test_config_get[args1-0-myremote]": 0.1829517000001033, "tests/func/test_config.py::test_config_get[args1-0-myremote]": 0.1829517000001033,
"tests/func/test_config.py::test_config_get[args2-0-iterative]": 0.19686669999998685, "tests/func/test_config.py::test_config_get[args2-0-treeverse]": 0.19686669999998685,
"tests/func/test_config.py::test_config_get[args3-0-iterative]": 0.1675059000000374, "tests/func/test_config.py::test_config_get[args3-0-treeverse]": 0.1675059000000374,
"tests/func/test_config.py::test_config_get[args4-251-option 'profile' doesn't exist]": 0.18096200000002227, "tests/func/test_config.py::test_config_get[args4-251-option 'profile' doesn't exist]": 0.18096200000002227,
"tests/func/test_config.py::test_config_get[args5-0-gs://bucket/path]": 0.12368970000022728, "tests/func/test_config.py::test_config_get[args5-0-gs://bucket/path]": 0.12368970000022728,
"tests/func/test_config.py::test_config_get[args6-0-gs://bucket/path]": 0.13007389999995667, "tests/func/test_config.py::test_config_get[args6-0-gs://bucket/path]": 0.13007389999995667,
@@ -2642,7 +2642,7 @@
"tests/unit/test_updater.py::test_is_enabled[config0-True]": 0.13130560000013247, "tests/unit/test_updater.py::test_is_enabled[config0-True]": 0.13130560000013247,
"tests/unit/test_updater.py::test_is_enabled[config1-True]": 0.14342600000009043, "tests/unit/test_updater.py::test_is_enabled[config1-True]": 0.14342600000009043,
"tests/unit/test_updater.py::test_is_enabled[config2-False]": 0.13593219999984285, "tests/unit/test_updater.py::test_is_enabled[config2-False]": 0.13593219999984285,
"tests/unit/test_updater.py::test_notify_message[None-Find the latest release at https://github.com/iterative/dvc/releases/latest.]": 0.014024399999925663, "tests/unit/test_updater.py::test_notify_message[None-Find the latest release at https://github.com/treeverse/dvc/releases/latest.]": 0.014024399999925663,
"tests/unit/test_updater.py::test_notify_message[binary-To upgrade, uninstall dvc and reinstall from https://dvc.org.]": 0.013217200000326557, "tests/unit/test_updater.py::test_notify_message[binary-To upgrade, uninstall dvc and reinstall from https://dvc.org.]": 0.013217200000326557,
"tests/unit/test_updater.py::test_notify_message[brew-To upgrade, run 'brew upgrade dvc'.]": 0.013467300000229443, "tests/unit/test_updater.py::test_notify_message[brew-To upgrade, run 'brew upgrade dvc'.]": 0.013467300000229443,
"tests/unit/test_updater.py::test_notify_message[choco-To upgrade, run 'choco upgrade dvc'.]": 0.014645900000004985, "tests/unit/test_updater.py::test_notify_message[choco-To upgrade, run 'choco upgrade dvc'.]": 0.014645900000004985,
@@ -2652,7 +2652,7 @@
"tests/unit/test_updater.py::test_notify_message[osxpkg-To upgrade, uninstall dvc and reinstall from https://dvc.org.]": 0.013980499999888707, "tests/unit/test_updater.py::test_notify_message[osxpkg-To upgrade, uninstall dvc and reinstall from https://dvc.org.]": 0.013980499999888707,
"tests/unit/test_updater.py::test_notify_message[pip-To upgrade, run 'pip install --upgrade dvc'.]": 0.015155299999833005, "tests/unit/test_updater.py::test_notify_message[pip-To upgrade, run 'pip install --upgrade dvc'.]": 0.015155299999833005,
"tests/unit/test_updater.py::test_notify_message[rpm-To upgrade, run 'yum update dvc'.]": 0.014036500000202068, "tests/unit/test_updater.py::test_notify_message[rpm-To upgrade, run 'yum update dvc'.]": 0.014036500000202068,
"tests/unit/test_updater.py::test_notify_message[unknown-Find the latest release at https://github.com/iterative/dvc/releases/latest.]": 0.01609110000003966, "tests/unit/test_updater.py::test_notify_message[unknown-Find the latest release at https://github.com/treeverse/dvc/releases/latest.]": 0.01609110000003966,
"tests/unit/ui/test_console.py::test_capsys_works": 0.003642499999841675, "tests/unit/ui/test_console.py::test_capsys_works": 0.003642499999841675,
"tests/unit/ui/test_console.py::test_write": 0.004320600000255581, "tests/unit/ui/test_console.py::test_write": 0.004320600000255581,
"tests/unit/ui/test_console.py::test_write_json[False-{\"hello\": \"world\", \"date\": \"1970-01-01 00:00:00\"}\\n]": 0.004433300000073359, "tests/unit/ui/test_console.py::test_write_json[False-{\"hello\": \"world\", \"date\": \"1970-01-01 00:00:00\"}\\n]": 0.004433300000073359,
+1 -1
View File
@@ -57,7 +57,7 @@ $ dvc doctor
**Additional Information (if any):** **Additional Information (if any):**
<!-- <!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue. Please check https://github.com/treeverse/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.
If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`. If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons. If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
+1 -1
View File
@@ -1,5 +1,5 @@
* [ ] ❗ I have followed the [Contributing to DVC](https://dvc.org/doc/user-guide/contributing/core) checklist. * [ ] ❗ I have followed the [Contributing to DVC](https://dvc.org/doc/user-guide/contributing/core) checklist.
* [ ] 📖 If this PR requires [documentation](https://dvc.org/doc) updates, I have created a separate PR (or issue, at least) in [dvc.org](https://github.com/iterative/dvc.org) and linked it here. * [ ] 📖 If this PR requires [documentation](https://dvc.org/doc) updates, I have created a separate PR (or issue, at least) in [dvc.org](https://github.com/treeverse/dvc.org) and linked it here.
Thank you for the contribution - we'll try to review it as soon as possible. 🙏 Thank you for the contribution - we'll try to review it as soon as possible. 🙏
+4 -4
View File
@@ -16,14 +16,14 @@ jobs:
steps: steps:
- uses: actions/setup-python@v6 - uses: actions/setup-python@v6
with: with:
python-version: "3.13" python-version: "3.14"
- uses: actions/checkout@v5 - uses: actions/checkout@v6
with: with:
ref: ${{ github.event.pull_request.base.sha }} ref: ${{ github.event.pull_request.base.sha }}
fetch-depth: 0 fetch-depth: 0
- uses: astral-sh/setup-uv@v6 - uses: astral-sh/setup-uv@v7
with: with:
enable-cache: true enable-cache: true
cache-dependency-glob: pyproject.toml cache-dependency-glob: pyproject.toml
@@ -32,7 +32,7 @@ jobs:
- name: run benchmarks on base branch - name: run benchmarks on base branch
run: pytest --benchmark-autosave dvc/testing/benchmarks/ -k 'test_init or test_help' run: pytest --benchmark-autosave dvc/testing/benchmarks/ -k 'test_init or test_help'
- uses: actions/checkout@v5 - uses: actions/checkout@v6
with: with:
fetch-depth: 0 fetch-depth: 0
clean: false clean: false
+7 -7
View File
@@ -19,15 +19,15 @@ jobs:
name: Build & Verify package name: Build & Verify package
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v5 - uses: actions/checkout@v6
with: with:
fetch-depth: 0 fetch-depth: 0
- uses: actions/setup-python@v6 - uses: actions/setup-python@v6
with: with:
python-version: 3.13 python-version: "3.14"
- uses: astral-sh/setup-uv@v6 - uses: astral-sh/setup-uv@v7
- name: Omit local version for Test PyPI upload - name: Omit local version for Test PyPI upload
if: github.event_name == 'push' && github.ref == 'refs/heads/main' if: github.event_name == 'push' && github.ref == 'refs/heads/main'
@@ -41,7 +41,7 @@ jobs:
- name: Check dist - name: Check dist
run: uv tool run twine check --strict dist/* run: uv tool run twine check --strict dist/*
- uses: actions/upload-artifact@v4 - uses: actions/upload-artifact@v7
with: with:
name: Packages name: Packages
path: dist/ path: dist/
@@ -51,7 +51,7 @@ jobs:
name: Publish dev package to test.pypi.org name: Publish dev package to test.pypi.org
runs-on: ubuntu-latest runs-on: ubuntu-latest
needs: build needs: build
if: ${{ github.event.action == 'published' || (github.repository == 'iterative/dvc' && github.event_name == 'push' && github.ref == 'refs/heads/main') }} if: ${{ github.event.action == 'published' || (github.repository == 'treeverse/dvc' && github.event_name == 'push' && github.ref == 'refs/heads/main') }}
environment: environment:
name: test-pypi name: test-pypi
@@ -61,7 +61,7 @@ jobs:
id-token: write id-token: write
steps: steps:
- uses: actions/download-artifact@v5 - uses: actions/download-artifact@v8
with: with:
name: Packages name: Packages
path: dist path: dist
@@ -86,7 +86,7 @@ jobs:
id-token: write id-token: write
steps: steps:
- uses: actions/download-artifact@v5 - uses: actions/download-artifact@v8
with: with:
name: Packages name: Packages
path: dist path: dist
+3 -3
View File
@@ -28,13 +28,13 @@ jobs:
steps: steps:
- name: Checkout repository - name: Checkout repository
uses: actions/checkout@v5 uses: actions/checkout@v6
- name: Initialize CodeQL - name: Initialize CodeQL
uses: github/codeql-action/init@v3 uses: github/codeql-action/init@v4
with: with:
languages: ${{ matrix.language }} languages: ${{ matrix.language }}
queries: security-extended queries: security-extended
- name: Perform CodeQL Analysis - name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3 uses: github/codeql-action/analyze@v4
+5 -5
View File
@@ -19,17 +19,17 @@ jobs:
fail-fast: false fail-fast: false
matrix: matrix:
os: [ubuntu-latest] os: [ubuntu-latest]
pyv: ["3.13"] pyv: ["3.14"]
plugin: ["dvc-s3"] plugin: ["dvc-s3"]
steps: steps:
- uses: actions/checkout@v5 - uses: actions/checkout@v6
with: with:
path: dvc path: dvc
- uses: actions/checkout@v5 - uses: actions/checkout@v6
with: with:
repository: iterative/${{ matrix.plugin }} repository: treeverse/${{ matrix.plugin }}
ref: main ref: main
path: ${{ matrix.plugin }} path: ${{ matrix.plugin }}
@@ -38,7 +38,7 @@ jobs:
with: with:
python-version: ${{ matrix.pyv }} python-version: ${{ matrix.pyv }}
- uses: astral-sh/setup-uv@v6 - uses: astral-sh/setup-uv@v7
with: with:
enable-cache: true enable-cache: true
cache-dependency-glob: "**/pyproject.toml" cache-dependency-glob: "**/pyproject.toml"
+20 -8
View File
@@ -26,14 +26,14 @@ jobs:
matrix: matrix:
os: [ubuntu-latest, macos-latest, windows-latest] os: [ubuntu-latest, macos-latest, windows-latest]
steps: steps:
- uses: actions/checkout@v5 - uses: actions/checkout@v6
with: with:
fetch-depth: 0 fetch-depth: 0
- name: Set up Python - name: Set up Python
uses: actions/setup-python@v6 uses: actions/setup-python@v6
with: with:
python-version: '3.13' python-version: '3.14'
- uses: astral-sh/setup-uv@v6 - uses: astral-sh/setup-uv@v7
with: with:
enable-cache: true enable-cache: true
cache-dependency-glob: pyproject.toml cache-dependency-glob: pyproject.toml
@@ -42,7 +42,7 @@ jobs:
UV_LINK_MODE: ${{ runner.os == 'Windows' && 'symlink' || 'hardlink' }} UV_LINK_MODE: ${{ runner.os == 'Windows' && 'symlink' || 'hardlink' }}
- name: Cache mypy - name: Cache mypy
uses: actions/cache@v4 uses: actions/cache@v5
with: with:
path: .mypy_cache path: .mypy_cache
key: mypy-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }} key: mypy-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
@@ -55,7 +55,7 @@ jobs:
fail-fast: false fail-fast: false
matrix: matrix:
os: [ubuntu-latest, macos-latest] os: [ubuntu-latest, macos-latest]
pyv: ["3.9", "3.10", "3.11", "3.12", "3.13"] pyv: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
include: include:
- os: windows-latest - os: windows-latest
pyv: "3.9" pyv: "3.9"
@@ -117,15 +117,27 @@ jobs:
- os: windows-latest - os: windows-latest
pyv: "3.13" pyv: "3.13"
pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 4" pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 4"
- os: windows-latest
pyv: "3.14"
pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 1"
- os: windows-latest
pyv: "3.14"
pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 2"
- os: windows-latest
pyv: "3.14"
pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 3"
- os: windows-latest
pyv: "3.14"
pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 4"
steps: steps:
- uses: actions/checkout@v5 - uses: actions/checkout@v6
with: with:
fetch-depth: 0 fetch-depth: 0
- name: Set up Python - name: Set up Python
uses: actions/setup-python@v6 uses: actions/setup-python@v6
with: with:
python-version: ${{ matrix.pyv }} python-version: ${{ matrix.pyv }}
- uses: astral-sh/setup-uv@v6 - uses: astral-sh/setup-uv@v7
with: with:
enable-cache: true enable-cache: true
cache-dependency-glob: pyproject.toml cache-dependency-glob: pyproject.toml
@@ -143,7 +155,7 @@ jobs:
pytest ${{ matrix.pytestargs }} -n=logical --dist=worksteal --timeout=300 --durations=0 pytest ${{ matrix.pytestargs }} -n=logical --dist=worksteal --timeout=300 --durations=0
--cov --cov-report=xml --cov-report=term --durations-path=./.github/.test_durations --cov --cov-report=xml --cov-report=term --durations-path=./.github/.test_durations
- name: upload coverage report - name: upload coverage report
uses: codecov/codecov-action@v5 uses: codecov/codecov-action@v6
check: check:
if: always() if: always()
needs: [tests] needs: [tests]
+2 -2
View File
@@ -22,13 +22,13 @@ repos:
- id: sort-simple-yaml - id: sort-simple-yaml
- id: trailing-whitespace - id: trailing-whitespace
- repo: https://github.com/astral-sh/ruff-pre-commit - repo: https://github.com/astral-sh/ruff-pre-commit
rev: 'v0.13.2' rev: 'v0.15.8'
hooks: hooks:
- id: ruff-check - id: ruff-check
args: [--fix, --exit-non-zero-on-fix] args: [--fix, --exit-non-zero-on-fix]
- id: ruff-format - id: ruff-format
- repo: https://github.com/codespell-project/codespell - repo: https://github.com/codespell-project/codespell
rev: v2.4.1 rev: v2.4.2
hooks: hooks:
- id: codespell - id: codespell
additional_dependencies: ["tomli"] additional_dependencies: ["tomli"]
+1 -1
View File
@@ -4,5 +4,5 @@
"data-science", "data-version-control", "machine-learning", "git", "data-science", "data-version-control", "machine-learning", "git",
"developer-tools", "reproducibility", "collaboration", "ai", "python"], "developer-tools", "reproducibility", "collaboration", "ai", "python"],
"contributors": [ "contributors": [
{"name": "DVC team", "type": "Other", "affiliation": "Iterative"}] {"name": "DVC team", "type": "Other", "affiliation": "Treeverse"}]
} }
+2 -2
View File
@@ -6,8 +6,8 @@ message: >-
type: software type: software
authors: authors:
- name: The DVC team and contributors - name: The DVC team and contributors
website: 'https://github.com/iterative' website: 'https://github.com/treeverse'
repository-code: 'https://github.com/iterative/dvc' repository-code: 'https://github.com/treeverse/dvc'
url: 'https://dvc.org' url: 'https://dvc.org'
abstract: >- abstract: >-
DVC is a tool for data science that takes advantage of existing software engineering toolset. DVC is a tool for data science that takes advantage of existing software engineering toolset.
+1 -1
View File
@@ -186,7 +186,7 @@
same "printed page" as the copyright notice for easier same "printed page" as the copyright notice for easier
identification within third-party archives. identification within third-party archives.
Copyright 2017-2021 Iterative, Inc. Copyright 2025 Treeverse.
Licensed under the Apache License, Version 2.0 (the "License"); Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License. you may not use this file except in compliance with the License.
+9 -15
View File
@@ -167,7 +167,7 @@ To install the development version, run:
.. code-block:: bash .. code-block:: bash
pip install git+git://github.com/iterative/dvc pip install "dvc @ git+https://github.com/treeverse/dvc"
Package (Platform-specific) Package (Platform-specific)
--------------------------- ---------------------------
@@ -175,7 +175,7 @@ Package (Platform-specific)
|Packages| |Packages|
Self-contained packages for Linux, Windows, and Mac are available. Self-contained packages for Linux, Windows, and Mac are available.
The latest version of the packages can be found on the GitHub `releases page <https://github.com/iterative/dvc/releases>`_. The latest version of the packages can be found on the GitHub `releases page <https://github.com/treeverse/dvc/releases>`_.
Ubuntu / Debian (deb) Ubuntu / Debian (deb)
^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
@@ -198,8 +198,6 @@ Fedora / CentOS (rpm)
Contributing Contributing
============ ============
|Maintainability|
Contributions are welcome! Contributions are welcome!
Please see our `Contributing Guide <https://dvc.org/doc/user-guide/contributing/core>`_ for more details. Please see our `Contributing Guide <https://dvc.org/doc/user-guide/contributing/core>`_ for more details.
Thanks to all our contributors! Thanks to all our contributors!
@@ -237,23 +235,19 @@ Barrak, A., Eghan, E.E. and Adams, B. `On the Co-evolution of ML Pipelines and S
:target: https://dvc.org :target: https://dvc.org
:alt: DVC logo :alt: DVC logo
.. |VS Code Extension Overview| image:: https://raw.githubusercontent.com/iterative/vscode-dvc/main/extension/docs/overview.gif .. |VS Code Extension Overview| image:: https://raw.githubusercontent.com/treeverse/vscode-dvc/main/extension/docs/overview.gif
:alt: DVC Extension for VS Code :alt: DVC Extension for VS Code
.. |CI| image:: https://github.com/iterative/dvc/actions/workflows/tests.yaml/badge.svg .. |CI| image:: https://github.com/treeverse/dvc/actions/workflows/tests.yaml/badge.svg
:target: https://github.com/iterative/dvc/actions/workflows/tests.yaml :target: https://github.com/treeverse/dvc/actions/workflows/tests.yaml
:alt: GHA Tests :alt: GHA Tests
.. |Maintainability| image:: https://img.shields.io/codeclimate/maintainability/iterative/dvc
:target: https://codeclimate.com/github/iterative/dvc
:alt: Code Climate
.. |Python Version| image:: https://img.shields.io/pypi/pyversions/dvc .. |Python Version| image:: https://img.shields.io/pypi/pyversions/dvc
:target: https://pypi.org/project/dvc :target: https://pypi.org/project/dvc
:alt: Python Version :alt: Python Version
.. |Coverage| image:: https://codecov.io/gh/iterative/dvc/branch/main/graph/badge.svg .. |Coverage| image:: https://codecov.io/gh/treeverse/dvc/branch/main/graph/badge.svg
:target: https://codecov.io/gh/iterative/dvc :target: https://codecov.io/gh/treeverse/dvc
:alt: Codecov :alt: Codecov
.. |Snap| image:: https://img.shields.io/badge/snap-install-82BEA0.svg?logo=snapcraft .. |Snap| image:: https://img.shields.io/badge/snap-install-82BEA0.svg?logo=snapcraft
@@ -292,8 +286,8 @@ Barrak, A., Eghan, E.E. and Adams, B. `On the Co-evolution of ML Pipelines and S
:target: https://dvc.org/img/flow.gif :target: https://dvc.org/img/flow.gif
:alt: how_dvc_works :alt: how_dvc_works
.. |Contribs| image:: https://contrib.rocks/image?repo=iterative/dvc .. |Contribs| image:: https://contrib.rocks/image?repo=treeverse/dvc
:target: https://github.com/iterative/dvc/graphs/contributors :target: https://github.com/treeverse/dvc/graphs/contributors
:alt: Contributors :alt: Contributors
.. |VS Code| image:: https://img.shields.io/visual-studio-marketplace/v/Iterative.dvc?color=blue&label=VSCode&logo=visualstudiocode&logoColor=blue .. |VS Code| image:: https://img.shields.io/visual-studio-marketplace/v/Iterative.dvc?color=blue&label=VSCode&logo=visualstudiocode&logoColor=blue
+2 -2
View File
@@ -8,7 +8,7 @@ from PyInstaller.utils.hooks import collect_submodules, is_module_or_submodule
_EXCLUDES = ("celery.bin", "celery.contrib") _EXCLUDES = ("celery.bin", "celery.contrib")
hiddenimports = collect_submodules( hiddenimports = collect_submodules(
"celery", "celery",
filter=lambda name: not any( filter=lambda name: (
is_module_or_submodule(name, module) for module in _EXCLUDES not any(is_module_or_submodule(name, module) for module in _EXCLUDES)
), ),
) )
-1
View File
@@ -8,7 +8,6 @@ datas += copy_metadata("gcsfs")
datas += copy_metadata("pyarrow") datas += copy_metadata("pyarrow")
datas += copy_metadata("pydrive2") datas += copy_metadata("pydrive2")
datas += copy_metadata("s3fs", recursive=True) datas += copy_metadata("s3fs", recursive=True)
datas += copy_metadata("boto3")
datas += copy_metadata("ossfs") datas += copy_metadata("ossfs")
datas += copy_metadata("sshfs") datas += copy_metadata("sshfs")
datas += copy_metadata("webdav4") datas += copy_metadata("webdav4")
+1 -1
View File
@@ -206,7 +206,7 @@ def add_debugging_flags(parser):
from argparse import SUPPRESS from argparse import SUPPRESS
# For detailed info see: # For detailed info see:
# https://github.com/iterative/dvc/wiki/Debugging,-Profiling-and-Benchmarking-DVC # https://github.com/treeverse/dvc/wiki/Debugging,-Profiling-and-Benchmarking-DVC
args, _ = parser.parse_known_args() args, _ = parser.parse_known_args()
verbose = args.verbose verbose = args.verbose
+1 -1
View File
@@ -147,7 +147,7 @@ def _git_remote_path_hash(scm: Optional["Base"]) -> Optional[str]:
path = _parse_git_remote_path(remote_url) path = _parse_git_remote_path(remote_url)
h = hashlib.md5(path.encode("utf-8"), usedforsecurity=False) # for FIPS h = hashlib.md5(path.encode("utf-8"), usedforsecurity=False) # for FIPS
return h.hexdigest() return h.hexdigest()
except Exception: # noqa: BLE001 except Exception:
logger.debug("Failed to get git remote path", exc_info=True) logger.debug("Failed to get git remote path", exc_info=True)
return None return None
+1 -1
View File
@@ -96,7 +96,7 @@ def _log_exceptions(exc: Exception) -> Optional[int]:
f"See {link} for more info." f"See {link} for more info."
) )
else: else:
link = format_link("https://github.com/iterative/dvc/issues") link = format_link("https://github.com/treeverse/dvc/issues")
hint = f"\nPlease report this bug to {link}. Thank you!" hint = f"\nPlease report this bug to {link}. Thank you!"
logger.exception( logger.exception(
+71 -2
View File
@@ -127,19 +127,80 @@ def _filter(graph, targets, full):
return new_graph return new_graph
def _build(repo, target=None, full=False, outs=False): def _is_foreach_matrix_stage(node, join_string):
if node.endswith(".dvc"):
return False
return join_string in node
def _collapse_foreach_matrix_get_nodes(graph):
from dvc.parsing import JOIN
new_nodes = set()
nodes_to_remove = set()
for _node in list(graph.nodes):
if not _is_foreach_matrix_stage(_node, JOIN):
continue
nodes_to_remove.add(_node)
new_nodes.add(_node.split(JOIN)[0])
return new_nodes, nodes_to_remove
def _collapse_foreach_matrix_get_edges(graph):
from dvc.parsing import JOIN
new_edges = set()
edges_to_remove = set()
for _e1, _e2 in list(graph.edges):
_replace = False
_new_e1 = _e1
_new_e2 = _e2
if _is_foreach_matrix_stage(_e1, JOIN):
_new_e1 = _e1.split(JOIN)[0]
_replace = True
if _is_foreach_matrix_stage(_e2, JOIN):
_new_e2 = _e2.split(JOIN)[0]
_replace = True
if _replace:
edges_to_remove.add((_e1, _e2))
new_edges.add((_new_e1, _new_e2))
return new_edges, edges_to_remove
def _collapse_foreach_matrix(graph):
new_nodes, nodes_to_remove = _collapse_foreach_matrix_get_nodes(graph)
new_edges, edges_to_remove = _collapse_foreach_matrix_get_edges(graph)
new_graph = graph.copy()
new_graph.remove_edges_from(edges_to_remove)
new_graph.add_nodes_from(new_nodes)
new_graph.add_edges_from(new_edges)
new_graph.remove_nodes_from(nodes_to_remove)
return new_graph
def _build(repo, target=None, full=False, outs=False, collapse_foreach_matrix=False):
targets = _collect_targets(repo, target, outs) targets = _collect_targets(repo, target, outs)
graph = _transform(repo.index, outs) graph = _transform(repo.index, outs)
return _filter(graph, targets, full) filtered_graph = _filter(graph, targets, full)
if collapse_foreach_matrix:
filtered_graph = _collapse_foreach_matrix(filtered_graph)
return filtered_graph
class CmdDAG(CmdBase): class CmdDAG(CmdBase):
def run(self): def run(self):
from dvc.exceptions import InvalidArgumentError
if self.args.outs and self.args.collapse_foreach_matrix:
raise InvalidArgumentError(
"`--outs` and `--collapse-foreach-matrix` are mutually exclusive"
)
graph = _build( graph = _build(
self.repo, self.repo,
target=self.args.target, target=self.args.target,
full=self.args.full, full=self.args.full,
outs=self.args.outs, outs=self.args.outs,
collapse_foreach_matrix=self.args.collapse_foreach_matrix,
) )
if self.args.dot: if self.args.dot:
@@ -181,6 +242,14 @@ def add_parser(subparsers, parent_parser):
dest="markdown", dest="markdown",
help="Print DAG with mermaid format wrapped in Markdown block.", help="Print DAG with mermaid format wrapped in Markdown block.",
) )
dag_parser.add_argument(
"--collapse-foreach-matrix",
action="store_true",
default=False,
help=(
"Collapse stages from each foreach/matrix definition into a single node."
),
)
dag_parser.add_argument( dag_parser.add_argument(
"--full", "--full",
action="store_true", action="store_true",
+11
View File
@@ -20,6 +20,7 @@ class CmdExperimentsRun(CmdRepro):
tmp_dir=self.args.tmp_dir, tmp_dir=self.args.tmp_dir,
copy_paths=self.args.copy_paths, copy_paths=self.args.copy_paths,
message=self.args.message, message=self.args.message,
no_hydra=self.args.no_hydra,
**self._common_kwargs, **self._common_kwargs,
) )
@@ -107,4 +108,14 @@ def _add_run_common(parser):
default=None, default=None,
help="Custom commit message to use when committing the experiment.", help="Custom commit message to use when committing the experiment.",
) )
parser.add_argument(
"--no-hydra",
action="store_true",
default=False,
help=(
"Disables automatically updating `params.yaml` with Hydra configuration. "
" You can still use `--set-param` to update individual params if needed."
" Default is False."
),
)
parser.add_argument("-M", dest="message", help=argparse.SUPPRESS) # obsolete parser.add_argument("-M", dest="message", help=argparse.SUPPRESS) # obsolete
+11 -4
View File
@@ -40,9 +40,16 @@ class CmdGC(CmdBase):
msg += " and all git commits" msg += " and all git commits"
else: else:
if self.args.all_branches and self.args.all_tags: if self.args.all_branches and self.args.all_tags:
msg += " and all git branches and tags" if self.args.num:
msg += f" and last {self.args.num} commits from all git branches"
msg += " and all git tags"
else:
msg += " and all git branches and tags"
elif self.args.all_branches: elif self.args.all_branches:
msg += " and all git branches" if self.args.num:
msg += f" and last {self.args.num} commits from all git branches"
else:
msg += " and all git branches"
elif self.args.all_tags: elif self.args.all_tags:
msg += " and all git tags" msg += " and all git tags"
if self.args.commit_date: if self.args.commit_date:
@@ -126,8 +133,8 @@ def add_parser(subparsers, parent_parser):
metavar="<num>", metavar="<num>",
help=( help=(
"Keep data files used in the last `num` commits " "Keep data files used in the last `num` commits "
"starting from the `--rev` <commit>. " "starting from each selected revision root. "
"Only used if `--rev` is also provided. " "Can be used with `--rev` and `--all-branches`. "
"Defaults to `1`." "Defaults to `1`."
), ),
) )
+1 -1
View File
@@ -29,7 +29,7 @@ def _welcome_message():
"{yellow}------------{nc}\n" "{yellow}------------{nc}\n"
f"- Check out the documentation: {fmt_link('https://dvc.org/doc')}\n" f"- Check out the documentation: {fmt_link('https://dvc.org/doc')}\n"
f"- Get help and share ideas: {fmt_link('https://dvc.org/chat')}\n" f"- Get help and share ideas: {fmt_link('https://dvc.org/chat')}\n"
f"- Star us on GitHub: {fmt_link('https://github.com/iterative/dvc')}" f"- Star us on GitHub: {fmt_link('https://github.com/treeverse/dvc')}"
).format(yellow=colorama.Fore.YELLOW, nc=colorama.Fore.RESET) ).format(yellow=colorama.Fore.YELLOW, nc=colorama.Fore.RESET)
ui.write(msg) ui.write(msg)
+1 -9
View File
@@ -109,8 +109,6 @@ def _build_tree_structure(
def show_tree(entries, with_color=False, with_size=False, with_hash=False): def show_tree(entries, with_color=False, with_size=False, with_hash=False):
import tabulate
rows = _build_tree_structure( rows = _build_tree_structure(
entries, entries,
with_color=with_color, with_color=with_color,
@@ -119,13 +117,7 @@ def show_tree(entries, with_color=False, with_size=False, with_hash=False):
) )
colalign = ("right",) if with_size else None colalign = ("right",) if with_size else None
ui.table(rows, colalign=colalign, preserve_whitespace=True)
_orig = tabulate.PRESERVE_WHITESPACE
tabulate.PRESERVE_WHITESPACE = True
try:
ui.table(rows, colalign=colalign)
finally:
tabulate.PRESERVE_WHITESPACE = _orig
class CmdList(CmdBaseNoRepo): class CmdList(CmdBaseNoRepo):
-12
View File
@@ -41,18 +41,6 @@ class RemoteNotFoundError(RemoteConfigError):
pass pass
class MachineConfigError(ConfigError):
pass
class NoMachineError(MachineConfigError):
pass
class MachineNotFoundError(MachineConfigError):
pass
@memoize @memoize
def get_compiled_schema(): def get_compiled_schema():
from voluptuous import Schema from voluptuous import Schema
+1
View File
@@ -151,6 +151,7 @@ WEBDAV_COMMON = {
"password": str, "password": str,
"ask_password": Bool, "ask_password": Bool,
"token": str, "token": str,
"bearer_token_command": str,
"custom_auth_header": str, "custom_auth_header": str,
"cert_path": str, "cert_path": str,
"key_path": str, "key_path": str,
+1 -1
View File
@@ -41,7 +41,7 @@ def _get(stage, p, info, **kwargs):
return Dependency(stage, p, info, **kwargs) return Dependency(stage, p, info, **kwargs)
def loadd_from(stage, d_list): def loadd_from(stage, d_list) -> list[Dependency]:
ret = [] ret = []
for d in d_list: for d in d_list:
p = d.pop(Output.PARAM_PATH, None) p = d.pop(Output.PARAM_PATH, None)
+1
View File
@@ -110,6 +110,7 @@ class ParamsDependency(Dependency):
except MissingParamsFile: except MissingParamsFile:
return {} return {}
assert self.repo
try: try:
return read_param_file( return read_param_file(
self.repo.fs, self.repo.fs,
+1
View File
@@ -150,6 +150,7 @@ class RepoDependency(Dependency):
# local cache instead of the remote's cache. This avoids re-streaming of data, # local cache instead of the remote's cache. This avoids re-streaming of data,
# but messes up the call to `_get_remote_config()` downstream, which will need # but messes up the call to `_get_remote_config()` downstream, which will need
# to ignore cache parameters. # to ignore cache parameters.
assert self.repo
config["cache"] = self.repo.config["cache"] config["cache"] = self.repo.config["cache"]
config["cache"]["dir"] = self.repo.cache.local_cache_dir config["cache"]["dir"] = self.repo.cache.local_cache_dir
+20 -5
View File
@@ -5,7 +5,15 @@ from collections.abc import Iterable, Iterator
from itertools import chain, groupby, takewhile from itertools import chain, groupby, takewhile
from typing import TYPE_CHECKING, Any, Literal, NamedTuple, Optional, Union, overload from typing import TYPE_CHECKING, Any, Literal, NamedTuple, Optional, Union, overload
from pathspec.patterns import GitWildMatchPattern try:
from pathspec.patterns.gitignore.spec import ( # type: ignore[import-not-found]
GitIgnoreSpecPattern,
)
except ImportError: # pathspec<1
from pathspec.patterns import (
GitWildMatchPattern as GitIgnoreSpecPattern,
)
from pathspec.util import normalize_file from pathspec.util import normalize_file
from pygtrie import Trie from pygtrie import Trie
@@ -34,7 +42,14 @@ class DvcIgnorePatterns(DvcIgnore):
def __init__( def __init__(
self, pattern_list: Iterable[Union[PatternInfo, str]], dirname: str, sep: str self, pattern_list: Iterable[Union[PatternInfo, str]], dirname: str, sep: str
) -> None: ) -> None:
from pathspec.patterns.gitwildmatch import _DIR_MARK try:
from pathspec.patterns.gitignore.spec import ( # type: ignore[import-not-found]
_DIR_MARK,
)
except ImportError: # pathspec<1
from pathspec.patterns.gitwildmatch import ( # type: ignore[attr-defined, no-redef]
_DIR_MARK,
)
pattern_infos = [ pattern_infos = [
pattern if isinstance(pattern, PatternInfo) else PatternInfo(pattern, "") pattern if isinstance(pattern, PatternInfo) else PatternInfo(pattern, "")
@@ -48,7 +63,7 @@ class DvcIgnorePatterns(DvcIgnore):
regex_pattern_list: list[tuple[str, bool, bool, PatternInfo]] = [] regex_pattern_list: list[tuple[str, bool, bool, PatternInfo]] = []
for count, pattern_info in enumerate(pattern_infos): for count, pattern_info in enumerate(pattern_infos):
regex, ignore = GitWildMatchPattern.pattern_to_regex(pattern_info.patterns) regex, ignore = GitIgnoreSpecPattern.pattern_to_regex(pattern_info.patterns)
if regex is not None and ignore is not None: if regex is not None and ignore is not None:
self.pattern_list.append(pattern_info) self.pattern_list.append(pattern_info)
regex = regex.replace(f"<{_DIR_MARK}>", f"<{_DIR_MARK}{count}>") regex = regex.replace(f"<{_DIR_MARK}>", f"<{_DIR_MARK}{count}>")
@@ -508,7 +523,7 @@ class DvcIgnoreFilter:
def check_ignore(self, target: str) -> CheckIgnoreResult: def check_ignore(self, target: str) -> CheckIgnoreResult:
# NOTE: can only be used in `dvc check-ignore`, see # NOTE: can only be used in `dvc check-ignore`, see
# https://github.com/iterative/dvc/issues/5046 # https://github.com/treeverse/dvc/issues/5046
full_target = self.fs.abspath(target) full_target = self.fs.abspath(target)
matched_patterns: list[PatternInfo] = [] matched_patterns: list[PatternInfo] = []
ignore = False ignore = False
@@ -525,7 +540,7 @@ class DvcIgnoreFilter:
self, fs: "FileSystem", path: str, ignore_subrepos: bool = True self, fs: "FileSystem", path: str, ignore_subrepos: bool = True
) -> bool: ) -> bool:
# NOTE: can't use self.check_ignore(path).match for now, see # NOTE: can't use self.check_ignore(path).match for now, see
# https://github.com/iterative/dvc/issues/4555 # https://github.com/treeverse/dvc/issues/4555
if fs.protocol != Schemes.LOCAL: if fs.protocol != Schemes.LOCAL:
return False return False
if fs.isfile(path): if fs.isfile(path):
+1 -1
View File
@@ -158,7 +158,7 @@ def _get_fs_type(path):
except PermissionError: except PermissionError:
pass pass
# need to follow the symlink: https://github.com/iterative/dvc/issues/5065 # need to follow the symlink: https://github.com/treeverse/dvc/issues/5065
path = pathlib.Path(path).resolve() path = pathlib.Path(path).resolve()
for parent in itertools.chain([path], path.parents): for parent in itertools.chain([path], path.parents):
+1 -1
View File
@@ -182,7 +182,7 @@ class HardlinkLock(flufl.lock.Lock, LockBase):
# number to generate the resulting lock file name, which is unique # number to generate the resulting lock file name, which is unique
# enough for our application. # enough for our application.
# #
# [1] https://github.com/iterative/dvc/issues/2582 # [1] https://github.com/treeverse/dvc/issues/2582
self._hostname = socket.gethostname() self._hostname = socket.gethostname()
self._lifetime = timedelta(days=365) # Lock for good by default self._lifetime = timedelta(days=365) # Lock for good by default
-2
View File
@@ -1,8 +1,6 @@
"""Manages logging configuration for DVC repo.""" """Manages logging configuration for DVC repo."""
import logging import logging
import logging.config
import logging.handlers
import os import os
import sys import sys
from collections.abc import Iterator from collections.abc import Iterator
+26 -9
View File
@@ -42,6 +42,7 @@ from .utils import relpath
from .utils.fs import path_isin from .utils.fs import path_isin
if TYPE_CHECKING: if TYPE_CHECKING:
from dvc.repo import Repo
from dvc_data.hashfile.obj import HashFile from dvc_data.hashfile.obj import HashFile
from dvc_data.index import DataIndexKey from dvc_data.index import DataIndexKey
@@ -83,7 +84,7 @@ def _get(stage, path, **kwargs):
return Output(stage, path, **kwargs) return Output(stage, path, **kwargs)
def loadd_from(stage, d_list): def loadd_from(stage, d_list) -> list["Output"]:
ret = [] ret = []
for d in d_list: for d in d_list:
p = d.pop(Output.PARAM_PATH) p = d.pop(Output.PARAM_PATH)
@@ -329,7 +330,7 @@ class Output:
self.annot = Annotation( self.annot = Annotation(
desc=desc, type=type, labels=labels or [], meta=meta or {} desc=desc, type=type, labels=labels or [], meta=meta or {}
) )
self.repo = stage.repo if not repo and stage else repo self.repo: Optional[Repo] = stage.repo if not repo and stage else repo
meta_d = merge_file_meta_from_cloud(info or {}) meta_d = merge_file_meta_from_cloud(info or {})
meta = Meta.from_dict(meta_d) meta = Meta.from_dict(meta_d)
# NOTE: when version_aware is not passed into get_cloud_fs, it will be # NOTE: when version_aware is not passed into get_cloud_fs, it will be
@@ -511,6 +512,7 @@ class Output:
from dvc.cachemgr import LEGACY_HASH_NAMES from dvc.cachemgr import LEGACY_HASH_NAMES
assert self.is_in_repo assert self.is_in_repo
assert self.repo
odb_name = "legacy" if self.hash_name in LEGACY_HASH_NAMES else "repo" odb_name = "legacy" if self.hash_name in LEGACY_HASH_NAMES else "repo"
return getattr(self.repo.cache, odb_name) return getattr(self.repo.cache, odb_name)
@@ -518,6 +520,7 @@ class Output:
def local_cache(self): def local_cache(self):
from dvc.cachemgr import LEGACY_HASH_NAMES from dvc.cachemgr import LEGACY_HASH_NAMES
assert self.repo
if self.hash_name in LEGACY_HASH_NAMES: if self.hash_name in LEGACY_HASH_NAMES:
return self.repo.cache.legacy return self.repo.cache.legacy
return self.repo.cache.local return self.repo.cache.local
@@ -585,6 +588,7 @@ class Output:
def index_key(self) -> tuple[str, "DataIndexKey"]: def index_key(self) -> tuple[str, "DataIndexKey"]:
if self.is_in_repo: if self.is_in_repo:
workspace = "repo" workspace = "repo"
assert self.repo
key = self.repo.fs.relparts(self.fs_path, self.repo.root_dir) key = self.repo.fs.relparts(self.fs_path, self.repo.root_dir)
else: else:
workspace = self.fs.protocol workspace = self.fs.protocol
@@ -640,6 +644,7 @@ class Output:
@property @property
def dvcignore(self) -> Optional["DvcIgnoreFilter"]: def dvcignore(self) -> Optional["DvcIgnoreFilter"]:
if self.fs.protocol == "local": if self.fs.protocol == "local":
assert self.repo
return self.repo.dvcignore return self.repo.dvcignore
return None return None
@@ -661,6 +666,7 @@ class Output:
if not self.use_scm_ignore: if not self.use_scm_ignore:
return return
assert self.repo
if self.repo.scm.is_tracked(self.fspath): if self.repo.scm.is_tracked(self.fspath):
raise OutputAlreadyTrackedError(self) raise OutputAlreadyTrackedError(self)
@@ -670,6 +676,7 @@ class Output:
if not self.use_scm_ignore: if not self.use_scm_ignore:
return return
assert self.repo
self.repo.scm_context.ignore_remove(self.fspath) self.repo.scm_context.ignore_remove(self.fspath)
def save(self) -> None: def save(self) -> None:
@@ -776,6 +783,7 @@ class Output:
callback=cb, callback=cb,
) )
if relink: if relink:
assert self.repo
rel = self.fs.relpath(filter_info or self.fs_path) rel = self.fs.relpath(filter_info or self.fs_path)
with CheckoutCallback(desc=f"Checking out {rel}", unit="files") as cb: with CheckoutCallback(desc=f"Checking out {rel}", unit="files") as cb:
self._checkout( self._checkout(
@@ -957,6 +965,7 @@ class Output:
added = not self.exists added = not self.exists
try: try:
assert self.repo
modified = self._checkout( modified = self._checkout(
filter_info or self.fs_path, filter_info or self.fs_path,
self.fs, self.fs,
@@ -987,18 +996,24 @@ class Output:
if ignore_remove: if ignore_remove:
self.ignore_remove() self.ignore_remove()
def move(self, out): def move(self, out: "Output") -> None:
src_exists = self.exists
if src_exists:
self.fs.move(self.fs_path, out.fs_path)
else:
logger.warning("%r missing", self.fspath)
if self.protocol == "local" and self.use_scm_ignore: if self.protocol == "local" and self.use_scm_ignore:
assert self.repo
self.repo.scm_context.ignore_remove(self.fspath) self.repo.scm_context.ignore_remove(self.fspath)
self.fs.move(self.fs_path, out.fs_path)
self.def_path = out.def_path self.def_path = out.def_path
self.fs_path = out.fs_path self.fs_path = out.fs_path
self.save() try:
self.commit() self.save()
self.commit()
if self.protocol == "local" and self.use_scm_ignore: except self.DoesNotExistError:
self.repo.scm_context.ignore(self.fspath) self.ignore()
def transfer( def transfer(
self, source, odb=None, jobs=None, update=False, no_progress_bar=False self, source, odb=None, jobs=None, update=False, no_progress_bar=False
@@ -1079,6 +1094,7 @@ class Output:
if self.remote: if self.remote:
kwargs["remote"] = self.remote kwargs["remote"] = self.remote
with suppress(Exception): with suppress(Exception):
assert self.repo
self.repo.cloud.pull([obj.hash_info], **kwargs) self.repo.cloud.pull([obj.hash_info], **kwargs)
if self.obj: if self.obj:
@@ -1172,6 +1188,7 @@ class Output:
return {} return {}
if self.remote: if self.remote:
assert self.repo
remote_odb = self.repo.cloud.get_remote_odb( remote_odb = self.repo.cloud.get_remote_odb(
name=self.remote, hash_name=self.hash_name name=self.remote, hash_name=self.hash_name
) )
+3 -3
View File
@@ -37,13 +37,13 @@ KEYCRE = re.compile(
def get_parser(): def get_parser():
from pyparsing import CharsNotIn, ParserElement, Suppress, ZeroOrMore from pyparsing import CharsNotIn, ParserElement, Suppress, ZeroOrMore
ParserElement.enablePackrat() ParserElement.enable_packrat()
word = CharsNotIn(f"{PERIOD}{LBRACK}{RBRACK}") word = CharsNotIn(f"{PERIOD}{LBRACK}{RBRACK}")
idx = Suppress(LBRACK) + word + Suppress(RBRACK) idx = Suppress(LBRACK) + word + Suppress(RBRACK)
attr = Suppress(PERIOD) + word attr = Suppress(PERIOD) + word
parser = word + ZeroOrMore(attr ^ idx) parser = word + ZeroOrMore(attr ^ idx)
parser.setParseAction(PERIOD.join) parser.set_parse_action(PERIOD.join)
return parser return parser
@@ -178,7 +178,7 @@ def parse_expr(s: str):
from pyparsing import ParseException from pyparsing import ParseException
try: try:
result = get_parser().parseString(s, parseAll=True) result = get_parser().parse_string(s, parse_all=True)
except ParseException as exc: except ParseException as exc:
format_and_raise_parse_error(exc) format_and_raise_parse_error(exc)
raise AssertionError("unreachable") # noqa: B904 raise AssertionError("unreachable") # noqa: B904
-1
View File
@@ -2,7 +2,6 @@ import os
from collections import defaultdict from collections import defaultdict
from typing import TYPE_CHECKING, NamedTuple, Optional from typing import TYPE_CHECKING, NamedTuple, Optional
import dpath
import dpath.options import dpath.options
from funcy import get_in, last from funcy import get_in, last
+4 -1
View File
@@ -20,6 +20,7 @@ if TYPE_CHECKING:
from dvc.fs.data import DataFileSystem from dvc.fs.data import DataFileSystem
from dvc.fs.dvc import DVCFileSystem from dvc.fs.dvc import DVCFileSystem
from dvc.lock import LockBase from dvc.lock import LockBase
from dvc.output import Output
from dvc.scm import Git, NoSCM from dvc.scm import Git, NoSCM
from dvc.stage import Stage from dvc.stage import Stage
from dvc.types import DictStrAny from dvc.types import DictStrAny
@@ -544,7 +545,9 @@ class Repo:
return used return used
def find_outs_by_path(self, path, outs=None, recursive=False, strict=True): def find_outs_by_path(
self, path, outs=None, recursive=False, strict=True
) -> list["Output"]:
# using `outs_graph` to ensure graph checks are run # using `outs_graph` to ensure graph checks are run
outs = outs or self.index.outs_graph outs = outs or self.index.outs_graph
+1 -1
View File
@@ -40,7 +40,7 @@ def check_name_format(name: str) -> None:
def name_is_compatible(name: str) -> bool: def name_is_compatible(name: str) -> bool:
""" """
Only needed by DVCLive per iterative/dvclive#715 Only needed by DVCLive per treeverse/dvclive#715
Will be removed in future release. Will be removed in future release.
""" """
from gto.constants import assert_name_is_valid from gto.constants import assert_name_is_valid
+27 -5
View File
@@ -1,6 +1,6 @@
import os import os
import posixpath import posixpath
from collections import defaultdict from collections import defaultdict, deque
from collections.abc import Iterable, Iterator, Mapping from collections.abc import Iterable, Iterator, Mapping
from typing import TYPE_CHECKING, Optional, TypedDict, Union from typing import TYPE_CHECKING, Optional, TypedDict, Union
@@ -385,6 +385,26 @@ def _transform_git_paths_to_dvc(repo: "Repo", files: Iterable[str]) -> list[str]
return [repo.fs.relpath(file, start) for file in files] return [repo.fs.relpath(file, start) for file in files]
def iter_index(
index: Union["BaseDataIndex", "DataIndexView"], shallow: bool = False
) -> Iterator[tuple["DataIndexKey", Optional["DataIndexEntry"]]]:
if not shallow:
yield from index.iteritems(shallow=shallow)
return
# only iterate until we find entries with hash_info in shallow mode
todo: deque[tuple[DataIndexKey, dict]] = deque([((), index.info(()))])
while todo:
key, info = todo.popleft()
entry = info.get("entry")
if info.get("type") == "directory" and not (entry and entry.hash_info):
try:
todo.extend(index.ls(key, detail=True))
except (KeyError, DataIndexDirError):
pass
yield key, entry
def _get_entries_not_in_remote( def _get_entries_not_in_remote(
repo: "Repo", repo: "Repo",
filter_keys: Optional[Iterable["DataIndexKey"]] = None, filter_keys: Optional[Iterable["DataIndexKey"]] = None,
@@ -418,7 +438,7 @@ def _get_entries_not_in_remote(
n = 0 n = 0
with TqdmCallback(size=n, desc="Checking remote", unit="entry") as cb: with TqdmCallback(size=n, desc="Checking remote", unit="entry") as cb:
for key, entry in view.iteritems(shallow=not granular): for key, entry in iter_index(view, shallow=not granular):
if not (entry and entry.hash_info): if not (entry and entry.hash_info):
continue continue
@@ -436,15 +456,17 @@ def _get_entries_not_in_remote(
n += 1 n += 1
cb.set_size(n) cb.set_size(n)
results = storage_map.bulk_remote_exists(
list(entries.values()), refresh=remote_refresh, callback=cb
)
for key, entry in entries.items(): for key, entry in entries.items():
k = (*key, "") if entry.meta and entry.meta.isdir else key k = (*key, "") if entry.meta and entry.meta.isdir else key
try: try:
if not storage_map.remote_exists(entry, refresh=remote_refresh): if not results.get(entry, False):
missing_entries.append(os.path.sep.join(k)) missing_entries.append(os.path.sep.join(k))
except StorageKeyError: except StorageKeyError:
pass pass
finally:
cb.relative_update()
data_index.onerror = orig_data_index_onerror data_index.onerror = orig_data_index_onerror
return missing_entries return missing_entries
+1 -1
View File
@@ -188,7 +188,7 @@ class Experiments:
rev_names = self.get_exact_name(revs) rev_names = self.get_exact_name(revs)
for rev in revs: for rev in revs:
name = rev_names[rev] name = rev_names[rev]
names.append(name if name else rev[:7]) names.append(name or rev[:7])
ui.write("\nRan experiment(s): {}".format(", ".join(names))) ui.write("\nRan experiment(s): {}".format(", ".join(names)))
if tmp_dir: if tmp_dir:
ui.write( ui.write(
+2 -2
View File
@@ -58,7 +58,7 @@ def collect_rev(
if not (rev == "workspace" or param_deps or data.contains_error): if not (rev == "workspace" or param_deps or data.contains_error):
cache.put(data, force=True) cache.put(data, force=True)
return ExpState(rev=rev, data=data) return ExpState(rev=rev, data=data)
except Exception as exc: # noqa: BLE001 except Exception as exc:
logger.debug("", exc_info=True) logger.debug("", exc_info=True)
error = SerializableError(str(exc), type(exc).__name__) error = SerializableError(str(exc), type(exc).__name__)
return ExpState(rev=rev, error=error) return ExpState(rev=rev, error=error)
@@ -328,7 +328,7 @@ def collect(
name=baseline_names.get(baseline_rev), name=baseline_names.get(baseline_rev),
data=baseline_data.data, data=baseline_data.data,
error=baseline_data.error, error=baseline_data.error,
experiments=experiments if experiments else None, experiments=experiments or None,
) )
) )
return result return result
+16 -4
View File
@@ -9,6 +9,7 @@ from enum import IntEnum
from itertools import chain from itertools import chain
from typing import TYPE_CHECKING, Any, Callable, NamedTuple, Optional, Union from typing import TYPE_CHECKING, Any, Callable, NamedTuple, Optional, Union
import funcy
from funcy import nullcontext from funcy import nullcontext
from scmrepo.exceptions import SCMError from scmrepo.exceptions import SCMError
@@ -756,7 +757,7 @@ class BaseExecutor(ABC):
logger.debug("Commit to current experiment branch '%s'", branch) logger.debug("Commit to current experiment branch '%s'", branch)
else: else:
baseline_rev = scm.get_ref(EXEC_BASELINE) baseline_rev = scm.get_ref(EXEC_BASELINE)
name = exp_name if exp_name else f"exp-{exp_hash[:5]}" name = exp_name or f"exp-{exp_hash[:5]}"
ref_info = ExpRefInfo(baseline_rev, name) ref_info = ExpRefInfo(baseline_rev, name)
branch = str(ref_info) branch = str(ref_info)
old_ref = None old_ref = None
@@ -815,11 +816,22 @@ class BaseExecutor(ABC):
@contextmanager @contextmanager
def set_temp_refs(self, scm: "Git", temp_dict: dict[str, str]): def set_temp_refs(self, scm: "Git", temp_dict: dict[str, str]):
# Retry ref set, get, and remove operations to handle transient issues during
# concurrent Git access.
# Dulwich deletes parent directories of refs if they happen to be empty after
# removing a ref, which can interfere with `set_ref` in other processes.
# `remove_ref` may also fail with a `FileLocked` error when refs are packed,
# since multiple processes might attempt to write to the same file.
retry = funcy.retry(10, errors=Exception, timeout=0.1)
set_ref = retry(scm.set_ref)
get_ref = retry(scm.get_ref)
remove_ref = retry(scm.remove_ref)
try: try:
for ref, rev in temp_dict.items(): for ref, rev in temp_dict.items():
scm.set_ref(ref, rev) set_ref(ref, rev)
yield yield
finally: finally:
for ref in temp_dict: for ref in temp_dict:
if scm.get_ref(ref): if get_ref(ref):
scm.remove_ref(ref) remove_ref(ref)
+6 -3
View File
@@ -287,6 +287,7 @@ class BaseStashQueue(ABC):
baseline_rev: Optional[str] = None, baseline_rev: Optional[str] = None,
branch: Optional[str] = None, branch: Optional[str] = None,
name: Optional[str] = None, name: Optional[str] = None,
no_hydra: bool = False,
**kwargs, **kwargs,
) -> QueueEntry: ) -> QueueEntry:
"""Stash changes from the workspace as an experiment. """Stash changes from the workspace as an experiment.
@@ -302,6 +303,7 @@ class BaseStashQueue(ABC):
name: Optional experiment name. If specified this will be used as name: Optional experiment name. If specified this will be used as
the human-readable name in the experiment branch ref. Has no the human-readable name in the experiment branch ref. Has no
effect of branch is specified. effect of branch is specified.
no_hydra: Disable Hydra from automatically overwriting all params.
.. _Hydra Override: .. _Hydra Override:
https://hydra.cc/docs/next/advanced/override_grammar/basic/ https://hydra.cc/docs/next/advanced/override_grammar/basic/
@@ -318,7 +320,7 @@ class BaseStashQueue(ABC):
# update experiment params from command line # update experiment params from command line
if params: if params:
self._update_params(params) self._update_params(params, no_hydra=no_hydra)
# DVC commit data deps to preserve state across workspace # DVC commit data deps to preserve state across workspace
# & tempdir runs # & tempdir runs
@@ -441,12 +443,13 @@ class BaseStashQueue(ABC):
f"from '{config_path}': {param_list}" f"from '{config_path}': {param_list}"
) )
def _update_params(self, params: dict[str, list[str]]): def _update_params(self, params: dict[str, list[str]], no_hydra: bool = False):
"""Update param files with the provided `Hydra Override`_ patterns. """Update param files with the provided `Hydra Override`_ patterns.
Args: Args:
params: Dict mapping paths to `Hydra Override`_ patterns, params: Dict mapping paths to `Hydra Override`_ patterns,
provided via `exp run --set-param`. provided via `exp run --set-param`.
no_hydra: Disable Hydra from automatically overwriting all params.
.. _Hydra Override: .. _Hydra Override:
https://hydra.cc/docs/advanced/override_grammar/basic/ https://hydra.cc/docs/advanced/override_grammar/basic/
@@ -456,7 +459,7 @@ class BaseStashQueue(ABC):
logger.debug("Using experiment params '%s'", params) logger.debug("Using experiment params '%s'", params)
hydra_config = self.repo.config.get("hydra", {}) hydra_config = self.repo.config.get("hydra", {})
hydra_enabled = hydra_config.get("enabled", False) hydra_enabled = hydra_config.get("enabled", False) and not no_hydra
hydra_output_file = ParamsDependency.DEFAULT_PARAMS_FILE hydra_output_file = ParamsDependency.DEFAULT_PARAMS_FILE
for path, overrides in params.items(): for path, overrides in params.items():
if hydra_enabled and path == hydra_output_file: if hydra_enabled and path == hydra_output_file:
+1 -1
View File
@@ -70,7 +70,7 @@ class LocalCeleryQueue(BaseStashQueue):
def celery(self) -> "FSApp": def celery(self) -> "FSApp":
from kombu.transport.filesystem import Channel from kombu.transport.filesystem import Channel
# related to https://github.com/iterative/dvc-task/issues/61 # related to https://github.com/treeverse/dvc-task/issues/61
Channel.QoS.restore_at_shutdown = False Channel.QoS.restore_at_shutdown = False
from dvc_task.app import FSApp from dvc_task.app import FSApp
+1 -1
View File
@@ -34,7 +34,7 @@ class ExpRefInfo:
def __init__(self, baseline_sha: str, name: Optional[str] = None): def __init__(self, baseline_sha: str, name: Optional[str] = None):
self.baseline_sha = baseline_sha self.baseline_sha = baseline_sha
self.name: str = name if name else "" self.name: str = name or ""
def __str__(self): def __str__(self):
return "/".join(self.parts) return "/".join(self.parts)
+4 -1
View File
@@ -22,6 +22,7 @@ def run( # noqa: C901, PLR0912
queue: bool = False, queue: bool = False,
copy_paths: Optional[Iterable[str]] = None, copy_paths: Optional[Iterable[str]] = None,
message: Optional[str] = None, message: Optional[str] = None,
no_hydra: bool = False,
**kwargs, **kwargs,
) -> dict[str, str]: ) -> dict[str, str]:
"""Reproduce the specified targets as an experiment. """Reproduce the specified targets as an experiment.
@@ -67,7 +68,7 @@ def run( # noqa: C901, PLR0912
else: else:
path_overrides = {} path_overrides = {}
hydra_enabled = repo.config.get("hydra", {}).get("enabled", False) hydra_enabled = repo.config.get("hydra", {}).get("enabled", False) and not no_hydra
hydra_output_file = ParamsDependency.DEFAULT_PARAMS_FILE hydra_output_file = ParamsDependency.DEFAULT_PARAMS_FILE
if hydra_enabled and hydra_output_file not in path_overrides: if hydra_enabled and hydra_output_file not in path_overrides:
# Force `_update_params` even if `--set-param` was not used # Force `_update_params` even if `--set-param` was not used
@@ -80,6 +81,7 @@ def run( # noqa: C901, PLR0912
tmp_dir=tmp_dir, tmp_dir=tmp_dir,
copy_paths=copy_paths, copy_paths=copy_paths,
message=message, message=message,
no_hydra=no_hydra,
**kwargs, **kwargs,
) )
@@ -100,6 +102,7 @@ def run( # noqa: C901, PLR0912
params=sweep_overrides, params=sweep_overrides,
copy_paths=copy_paths, copy_paths=copy_paths,
message=message, message=message,
no_hydra=no_hydra,
**kwargs, **kwargs,
) )
if sweep_overrides: if sweep_overrides:
+2 -4
View File
@@ -67,7 +67,7 @@ class ExpStash(Stash):
branch: Optional[str] = None, branch: Optional[str] = None,
) -> str: ) -> str:
msg = cls.MESSAGE_FORMAT.format( msg = cls.MESSAGE_FORMAT.format(
rev=rev, baseline_rev=baseline_rev, name=name if name else "" rev=rev, baseline_rev=baseline_rev, name=name or ""
) )
branch_msg = f":{branch}" if branch else "" branch_msg = f":{branch}" if branch else ""
return f"{msg}{branch_msg}" return f"{msg}{branch_msg}"
@@ -124,9 +124,7 @@ class ApplyStash(Stash):
@classmethod @classmethod
def format_message(cls, head_rev: str, rev: str, name: Optional[str] = None) -> str: def format_message(cls, head_rev: str, rev: str, name: Optional[str] = None) -> str:
return cls.MESSAGE_FORMAT.format( return cls.MESSAGE_FORMAT.format(head_rev=head_rev, rev=rev, name=name or "")
head_rev=head_rev, rev=rev, name=name if name else ""
)
@contextmanager @contextmanager
def preserve_workspace( def preserve_workspace(
+1 -1
View File
@@ -200,7 +200,7 @@ def exp_commits(
) -> Iterable[str]: ) -> Iterable[str]:
"""Iterate over all experiment commits.""" """Iterate over all experiment commits."""
shas: set[str] = set() shas: set[str] = set()
refs = ref_infos if ref_infos else exp_refs(scm) refs = ref_infos or exp_refs(scm)
for ref_info in refs: for ref_info in refs:
shas.update(scm.branch_revs(str(ref_info), ref_info.baseline_sha)) shas.update(scm.branch_revs(str(ref_info), ref_info.baseline_sha))
yield from shas yield from shas
+7 -3
View File
@@ -28,8 +28,10 @@ def _validate_args(**kwargs):
"`--all-experiments`, `--all-commits`, `--date` or `--rev` " "`--all-experiments`, `--all-commits`, `--date` or `--rev` "
"needs to be set." "needs to be set."
) )
if kwargs.get("num") and not kwargs.get("rev"): if kwargs.get("num") and not (kwargs.get("rev") or kwargs.get("all_branches")):
raise InvalidArgumentError("`--num` can only be used alongside `--rev`") raise InvalidArgumentError(
"`--num` can only be used alongside `--rev` or `--all-branches`"
)
def _used_obj_ids_not_in_remote( def _used_obj_ids_not_in_remote(
@@ -128,9 +130,11 @@ def gc( # noqa: C901, PLR0912, PLR0913
used_obj_ids = set() used_obj_ids = set()
used_obj_ids.update(*odb_to_obj_ids.values()) used_obj_ids.update(*odb_to_obj_ids.values())
seen_odbs = set()
for scheme, odb in self.cache.by_scheme(): for scheme, odb in self.cache.by_scheme():
if not odb: if not odb or odb in seen_odbs:
continue continue
seen_odbs.add(odb)
num_removed = ogc(odb, used_obj_ids, jobs=jobs, dry=dry) num_removed = ogc(odb, used_obj_ids, jobs=jobs, dry=dry)
if num_removed: if num_removed:
logger.info("Removed %d objects from %s cache.", num_removed, scheme) logger.info("Removed %d objects from %s cache.", num_removed, scheme)
+1 -1
View File
@@ -16,7 +16,7 @@ def pre_commit_install(scm: "Git") -> None:
config_path = os.path.join(scm.root_dir, ".pre-commit-config.yaml") config_path = os.path.join(scm.root_dir, ".pre-commit-config.yaml")
with modify_yaml(config_path) as config: with modify_yaml(config_path) as config:
entry = { entry = {
"repo": "https://github.com/iterative/dvc", "repo": "https://github.com/treeverse/dvc",
"rev": ".".join(map(str, version_tuple[:3])), "rev": ".".join(map(str, version_tuple[:3])),
"hooks": [ "hooks": [
{ {
+1 -1
View File
@@ -123,7 +123,7 @@ def try_expand_paths(fs: "FileSystem", paths: Iterable[str]) -> Iterator[str]:
if fs.isdir(path): if fs.isdir(path):
yield from fs.find(path) yield from fs.find(path)
continue continue
except Exception as e: # noqa: BLE001 except Exception as e:
logger.debug( logger.debug(
"failed to expand %r: %s", "failed to expand %r: %s",
path, path,
+41 -17
View File
@@ -1,10 +1,13 @@
import os import os
from typing import TYPE_CHECKING
from dvc.exceptions import MoveNotDataSourceError
from dvc.repo.scm_context import scm_context from dvc.repo.scm_context import scm_context
from . import locked from . import locked
if TYPE_CHECKING:
from . import Repo
def _expand_target_path(from_path, to_path): def _expand_target_path(from_path, to_path):
if os.path.isdir(to_path): if os.path.isdir(to_path):
@@ -14,7 +17,7 @@ def _expand_target_path(from_path, to_path):
@locked @locked
@scm_context @scm_context
def move(self, from_path, to_path): def move(self: "Repo", from_path, to_path):
""" """
Renames an output file and modifies the stage associated Renames an output file and modifies the stage associated
to reflect the change on the pipeline. to reflect the change on the pipeline.
@@ -32,9 +35,12 @@ def move(self, from_path, to_path):
It only works with outputs generated by `add` or `import`, It only works with outputs generated by `add` or `import`,
also known as data sources. also known as data sources.
""" """
from dvc import output from dvc import dependency, output
from dvc.dvcfile import DVC_FILE_SUFFIX from dvc.dvcfile import DVC_FILE_SUFFIX
from dvc.exceptions import MoveNotDataSourceError
from dvc.stage import Stage from dvc.stage import Stage
from dvc.stage.exceptions import StageFileAlreadyExistsError
from dvc_objects.fs.local import LocalFileSystem
from_out = output.loads_from(Stage(self), [from_path])[0] from_out = output.loads_from(Stage(self), [from_path])[0]
assert from_out.protocol == "local" assert from_out.protocol == "local"
@@ -45,6 +51,7 @@ def move(self, from_path, to_path):
assert len(outs) == 1 assert len(outs) == 1
out = outs[0] out = outs[0]
stage = out.stage stage = out.stage
deps = stage.deps
if not stage.is_data_source: if not stage.is_data_source:
raise MoveNotDataSourceError(stage.addressing) raise MoveNotDataSourceError(stage.addressing)
@@ -58,21 +65,38 @@ def move(self, from_path, to_path):
) )
new_wdir = os.path.abspath(os.path.join(os.curdir, os.path.dirname(to_path))) new_wdir = os.path.abspath(os.path.join(os.curdir, os.path.dirname(to_path)))
to_path = os.path.relpath(to_path, new_wdir) to_path = os.path.relpath(to_path, new_wdir)
new_stage = self.stage.create( try:
single_stage=True, new_stage = self.stage.create(
fname=new_fname, single_stage=True,
wdir=new_wdir, fname=new_fname,
outs=[to_path], wdir=new_wdir,
meta=stage.meta, outs=[to_path],
) meta=stage.meta,
frozen=stage.frozen,
os.unlink(stage.path) always_changed=stage.always_changed,
stage = new_stage desc=stage.desc,
)
except StageFileAlreadyExistsError:
# reraise to remove `--force` hint
raise StageFileAlreadyExistsError(f"{new_fname!r} already exists") from None
else: else:
new_stage = stage
to_path = os.path.relpath(to_path, stage.wdir) to_path = os.path.relpath(to_path, stage.wdir)
to_out = output.loads_from(stage, [to_path], out.use_cache, out.metric)[0] def with_dep_path_adjusted(dep: dependency.Dependency):
d = dep.dumpd()
if isinstance(dep.fs, LocalFileSystem) and not os.path.isabs(dep.def_path):
return d | {"path": os.path.relpath(dep.fspath, new_stage.wdir)}
return d
out.move(to_out) new_stage.outs = output.loadd_from(new_stage, [out.dumpd() | {"path": to_path}])
stage.save() new_stage.deps = dependency.loadd_from(
stage.dump() new_stage, [with_dep_path_adjusted(dep) for dep in deps]
)
out.move(new_stage.outs[0])
new_stage.md5 = new_stage.compute_md5()
new_stage.dump()
if stage != new_stage:
stage.dvcfile.remove()
self.scm_context.track_file(stage.dvcfile.relpath)
return stage, new_stage
+7 -4
View File
@@ -10,6 +10,8 @@ from dvc.utils import relpath
from dvc.utils.collections import ensure_list from dvc.utils.collections import ensure_list
if TYPE_CHECKING: if TYPE_CHECKING:
from typing_extensions import Self
from dvc.repo import Repo from dvc.repo import Repo
from dvc.scm import Base from dvc.scm import Base
@@ -46,7 +48,7 @@ class SCMContext:
try: try:
add_no_submodules(self.scm, paths) add_no_submodules(self.scm, paths)
except UnsupportedIndexFormat: except UnsupportedIndexFormat:
link = "https://github.com/iterative/dvc/issues/610" link = "https://github.com/treeverse/dvc/issues/610"
add_cmd = self._make_git_add_cmd([relpath(path) for path in paths]) add_cmd = self._make_git_add_cmd([relpath(path) for path in paths])
logger.info("") logger.info("")
msg = ( msg = (
@@ -96,7 +98,7 @@ class SCMContext:
@contextmanager @contextmanager
def __call__( def __call__(
self, autostage: Optional[bool] = None, quiet: Optional[bool] = None self, autostage: Optional[bool] = None, quiet: Optional[bool] = None
) -> Iterator["SCMContext"]: ) -> Iterator["Self"]:
try: try:
yield self yield self
except Exception: except Exception:
@@ -131,9 +133,10 @@ class SCMContext:
self.files_to_track = set() self.files_to_track = set()
def __enter__(self) -> "SCMContext": def __enter__(self) -> "Self":
self._cm = self() self._cm = self()
return self._cm.__enter__() self._cm.__enter__()
return self
def __exit__(self, *exc_args) -> None: def __exit__(self, *exc_args) -> None:
assert self._cm assert self._cm
+5 -2
View File
@@ -234,7 +234,7 @@ def iter_revs(
results.extend(scm.list_all_commits()) results.extend(scm.list_all_commits())
else: else:
if all_branches: if all_branches:
results.extend(scm.list_branches()) results.extend(_get_n_commits(scm, scm.list_branches(), num))
if all_tags: if all_tags:
results.extend(scm.list_tags()) results.extend(scm.list_tags())
@@ -258,7 +258,10 @@ def iter_revs(
results.extend(exp_commits(scm)) results.extend(exp_commits(scm))
rev_resolver = partial(resolve_rev, scm) rev_resolver = partial(resolve_rev, scm)
return group_by(rev_resolver, results) grouped = group_by(rev_resolver, results)
for rev, names in grouped.items():
grouped[rev] = list(dict.fromkeys(names))
return grouped
def lfs_prefetch(fs: "FileSystem", paths: list[str]): def lfs_prefetch(fs: "FileSystem", paths: list[str]):
+3 -3
View File
@@ -32,7 +32,7 @@ from .utils import (
) )
if TYPE_CHECKING: if TYPE_CHECKING:
from dvc.dependency import ParamsDependency from dvc.dependency import Dependency, ParamsDependency
from dvc.dvcfile import ProjectFile, SingleStageFile from dvc.dvcfile import ProjectFile, SingleStageFile
from dvc.output import Output from dvc.output import Output
from dvc.repo import Repo from dvc.repo import Repo
@@ -153,8 +153,8 @@ class Stage(params.StageParams):
self._path = path self._path = path
self.cmd = cmd self.cmd = cmd
self.wdir = wdir self.wdir = wdir
self.outs = outs self.outs: list[Output] = outs
self.deps = deps self.deps: list[Dependency] = deps
self.md5 = md5 self.md5 = md5
self.frozen = locked or frozen self.frozen = locked or frozen
self.always_changed = always_changed self.always_changed = always_changed
+2 -2
View File
@@ -51,7 +51,7 @@ def _warn_if_fish(executable):
"Be aware that it might cause problems by overwriting " "Be aware that it might cause problems by overwriting "
"your current environment variables with values defined " "your current environment variables with values defined "
"in 'config.fish', which might affect your command. See " "in 'config.fish', which might affect your command. See "
"https://github.com/iterative/dvc/issues/1307. " "https://github.com/treeverse/dvc/issues/1307. "
) )
@@ -99,7 +99,7 @@ def prepare_kwargs(stage, run_env=None):
# #
# [1] https://github.com/python/cpython/blob/3.7/Lib/subprocess.py # [1] https://github.com/python/cpython/blob/3.7/Lib/subprocess.py
# #L1426 # #L1426
# [2] https://github.com/iterative/dvc/issues/2506 # [2] https://github.com/treeverse/dvc/issues/2506
# #issuecomment-535396799 # #issuecomment-535396799
kwargs["shell"] = os.name == "nt" kwargs["shell"] = os.name == "nt"
return kwargs return kwargs
+1 -1
View File
@@ -178,7 +178,7 @@ def to_single_stage_lockfile(stage: "Stage", **kwargs) -> dict:
res = OrderedDict([("cmd", stage.cmd)]) res = OrderedDict([("cmd", stage.cmd)])
params, deps = split_params_deps(stage) params, deps = split_params_deps(stage)
deps, outs = ( deps, outs = (
[_dumpd(item) for item in sorted(items, key=attrgetter("def_path"))] [_dumpd(item) for item in sorted(items, key=attrgetter("def_path"))] # type: ignore[call-overload]
for items in [deps, stage.outs] for items in [deps, stage.outs]
) )
params = _serialize_params_values(params) params = _serialize_params_values(params)
+1 -1
View File
@@ -4,7 +4,7 @@ dvc.testing.benchmarks
====================== ======================
Benchmark test definitions are now part of ``dvc.testing``. Benchmark test definitions are now part of ``dvc.testing``.
For CLI usage and `bench.dvc.org <https://bench.dvc.org>`_ details see `dvc-bench <https://github.com/iterative/dvc-bench>`_. For CLI usage and `bench.dvc.org <https://bench.dvc.org>`_ details see `dvc-bench <https://github.com/treeverse/dvc-bench>`_.
``dvc.testing.benchmarks`` structure: ``dvc.testing.benchmarks`` structure:
+1 -1
View File
@@ -2,6 +2,7 @@ import pytest
from dvc import api from dvc import api
from dvc.api import DVCFileSystem from dvc.api import DVCFileSystem
from dvc.testing import matchers as M
from dvc.utils.fs import remove from dvc.utils.fs import remove
@@ -42,7 +43,6 @@ class TestAPI:
) )
def test_filesystem( def test_filesystem(
self, self,
M,
tmp_dir, tmp_dir,
make_tmp_dir, make_tmp_dir,
scm, scm,
@@ -6,8 +6,8 @@ import pytest
"!=3.53.*,!=3.54.0", "!=3.53.*,!=3.54.0",
reason="Takes 10 mins to run. Regression in 3.53.0, fixed in 3.54.1", reason="Takes 10 mins to run. Regression in 3.53.0, fixed in 3.54.1",
) )
# Introduced in https://github.com/iterative/dvc/pull/10388. # Introduced in https://github.com/treeverse/dvc/pull/10388.
# Fixed in https://github.com/iterative/dvc/pull/10531. # Fixed in https://github.com/treeverse/dvc/pull/10531.
def test_import(bench_dvc, tmp_dir, scm, dvc, make_dataset, remote): def test_import(bench_dvc, tmp_dir, scm, dvc, make_dataset, remote):
dataset = make_dataset( dataset = make_dataset(
cache=False, files=False, dvcfile=True, commit=True, remote=True cache=False, files=False, dvcfile=True, commit=True, remote=True
+17 -8
View File
@@ -9,7 +9,7 @@ from typing import Optional
import pytest import pytest
from dulwich.porcelain import clone from dulwich.porcelain import clone
from funcy import first from funcy import first
from packaging import version from packaging import specifiers, version
from dvc.types import StrPath from dvc.types import StrPath
@@ -79,7 +79,7 @@ def dvc_bench_repo(tmp_path_factory, bench_config):
if url is None: if url is None:
pytest.skip( pytest.skip(
"--dvc-bench-repo is not set, " "--dvc-bench-repo is not set, "
"clone https://github.com/iterative/dvc-bench repository and set its path" "clone https://github.com/treeverse/dvc-bench repository and set its path"
) )
if os.path.isdir(url): if os.path.isdir(url):
@@ -109,11 +109,20 @@ def make_dvc_bin(
else: else:
pkg = "dvc" pkg = "dvc"
packages = [f"{pkg} @ git+file://{dvc_repo}@{dvc_rev}"] packages = [f"{pkg} @ git+file://{dvc_repo}@{dvc_rev}"]
try:
if version.Version(dvc_rev) < version.Version("3.50.3"): version_constraints = [
packages.append("pygit2==1.14.1") ("<3.50.3", ["pygit2==1.14.1"]),
except version.InvalidVersion: ("<3.44.0", ["dulwich<1.0.0"]),
pass ("<3.67.0", ["pathspec<1"]),
]
for spec, pkgs in version_constraints:
try:
_dvc_version = version.Version(dvc_rev)
except version.InvalidVersion:
continue
if _dvc_version in specifiers.SpecifierSet(spec):
packages.extend(pkgs)
venv.install(*packages) venv.install(*packages)
dvc_venvs[dvc_rev] = venv dvc_venvs[dvc_rev] = venv
@@ -158,7 +167,7 @@ def make_bench(request):
# pytest >= 8.4.0 # pytest >= 8.4.0
wrapped_func = fixture_function._get_wrapped_function() wrapped_func = fixture_function._get_wrapped_function()
except AttributeError: except AttributeError:
wrapped_func = fixture_function.__pytest_wrapped__.obj # ty: ignore[unresolved-attribute] wrapped_func = fixture_function.__pytest_wrapped__.obj # type: ignore[attr-defined] # ty: ignore[unresolved-attribute]
assert inspect.isgeneratorfunction(wrapped_func) assert inspect.isgeneratorfunction(wrapped_func)
generator = wrapped_func(request) generator = wrapped_func(request)
+104
View File
@@ -0,0 +1,104 @@
from collections.abc import Iterable, Mapping
from typing import TYPE_CHECKING, Any, Optional, Union
if TYPE_CHECKING:
import builtins
class dict: # noqa: A001, N801, PLW1641
"""Special class to eq by matching only presented dict keys.
Implementation notes:
- can't inherit from dict because that makes D() == M.dict() to not call
our __eq__, if D is a subclass of a dict
- should not call itself dict or use dict in repr because it creates
confusing error messages (shadowing python builtins is bad anyway)
"""
def __init__(self, d: Optional[Mapping[Any, Any]] = None, **keys: Any) -> None:
self.d: builtins.dict[Any, Any] = {}
if d:
self.d.update(d)
self.d.update(keys)
def __len__(self) -> int:
return len(self.d)
def __repr__(self) -> str:
inner = ", ".join(f"{k}={v!r}" for k, v in self.d.items())
return f"{self.__class__.__name__}({inner})"
def __eq__(self, other: object) -> bool:
assert isinstance(other, Mapping)
return all(other.get(name) == v for name, v in self.d.items())
class unordered: # noqa: N801, PLW1641
"""Compare list contents, but do not care about ordering.
(E.g. sort lists first, then compare.)
If you care about ordering, then just compare lists directly."""
def __init__(self, *items: Any) -> None:
self.items = items
def __repr__(self) -> str:
inner = ", ".join(map(repr, self.items))
return f"{self.__class__.__name__}({inner})"
def __eq__(self, other: object) -> bool:
assert isinstance(other, Iterable)
return sorted(self.items) == sorted(other)
class attrs: # noqa: N801, PLW1641
def __init__(self, **attribs: Any) -> None:
self.attribs = attribs
def __repr__(self) -> str:
inner = ", ".join(f"{k}={v!r}" for k, v in self.attribs.items())
return f"{self.__class__.__name__}({inner})"
def __eq__(self, other: object) -> bool:
# Unforturnately this doesn't work with classes with slots
# self.__class__ = other.__class__
return all(getattr(other, name) == v for name, v in self.attribs.items())
class instance_of: # noqa: N801, PLW1641
def __init__(self, expected_type: Union[Any, tuple[Any, ...]]) -> None:
self.expected_type = expected_type
def __repr__(self) -> str:
if isinstance(self.expected_type, tuple):
inner = f"({', '.join(t.__name__ for t in self.expected_type)})"
else:
inner = self.expected_type.__name__
return f"{self.__class__.__name__}({inner})"
def __eq__(self, other: object) -> bool:
return isinstance(other, self.expected_type)
class any_of: # noqa: N801, PLW1641
def __init__(self, *items: Any) -> None:
self.items = sorted(items)
def __repr__(self) -> str:
inner = ", ".join(map(repr, self.items))
return f"any_of({inner})"
def __eq__(self, other: object) -> bool:
return other in self.items
__all__ = [
"any_of",
"attrs",
"dict",
"instance_of",
"unordered",
]
+6 -4
View File
@@ -2,7 +2,7 @@ import os
import pathlib import pathlib
import posixpath import posixpath
import sys import sys
from typing import Callable, ClassVar from typing import Callable, ClassVar, Union
from urllib.parse import urlparse from urllib.parse import urlparse
from dvc.utils import relpath from dvc.utils import relpath
@@ -79,15 +79,17 @@ class PathInfo(pathlib.PurePath, _BasePath):
and self._cparts[:n] == other._cparts # type: ignore[attr-defined] and self._cparts[:n] == other._cparts # type: ignore[attr-defined]
) )
def relative_to(self, other, *args, **kwargs): def relative_to(
self, *other: Union[str, "os.PathLike[str]"], **kwargs
) -> "PathInfo":
# pathlib relative_to raises exception when one path is not a direct # pathlib relative_to raises exception when one path is not a direct
# descendant of the other when os.path.relpath would return abspath. # descendant of the other when os.path.relpath would return abspath.
# For DVC PathInfo we only need the relpath behavior. # For DVC PathInfo we only need the relpath behavior.
# See: https://bugs.python.org/issue40358 # See: https://bugs.python.org/issue40358
try: try:
path = super().relative_to(other, *args, **kwargs) path = super().relative_to(*other, **kwargs)
except ValueError: except ValueError:
path = relpath(self, other) path = relpath(self, other[0])
return self.__class__(path) return self.__class__(path)
+3 -3
View File
@@ -44,7 +44,7 @@ class TestRemote:
_check_status(status_dir, new=dir_hashes) _check_status(status_dir, new=dir_hashes)
# Move cache and check status # Move cache and check status
# See issue https://github.com/iterative/dvc/issues/4383 for details # See issue https://github.com/treeverse/dvc/issues/4383 for details
backup_dir = dvc.cache.local.path + ".backup" backup_dir = dvc.cache.local.path + ".backup"
shutil.move(dvc.cache.local.path, backup_dir) shutil.move(dvc.cache.local.path, backup_dir)
status = dvc.cloud.status(foo_hashes) status = dvc.cloud.status(foo_hashes)
@@ -116,7 +116,7 @@ class TestRemote:
@pytest.mark.xfail(raises=NotImplementedError, strict=False) @pytest.mark.xfail(raises=NotImplementedError, strict=False)
def test_pull_00_prefix(self, tmp_dir, dvc, remote, monkeypatch): def test_pull_00_prefix(self, tmp_dir, dvc, remote, monkeypatch):
# Related: https://github.com/iterative/dvc/issues/6089 # Related: https://github.com/treeverse/dvc/issues/6089
fs_type = type(dvc.cloud.get_remote_odb("upstream").fs) fs_type = type(dvc.cloud.get_remote_odb("upstream").fs)
monkeypatch.setattr(fs_type, "_ALWAYS_TRAVERSE", True, raising=False) monkeypatch.setattr(fs_type, "_ALWAYS_TRAVERSE", True, raising=False)
@@ -146,7 +146,7 @@ class TestRemote:
@pytest.mark.xfail(raises=NotImplementedError, strict=False) @pytest.mark.xfail(raises=NotImplementedError, strict=False)
def test_pull_no_00_prefix(self, tmp_dir, dvc, remote, monkeypatch): def test_pull_no_00_prefix(self, tmp_dir, dvc, remote, monkeypatch):
# Related: https://github.com/iterative/dvc/issues/6244 # Related: https://github.com/treeverse/dvc/issues/6244
fs_type = type(dvc.cloud.get_remote_odb("upstream").fs) fs_type = type(dvc.cloud.get_remote_odb("upstream").fs)
monkeypatch.setattr(fs_type, "_ALWAYS_TRAVERSE", True, raising=False) monkeypatch.setattr(fs_type, "_ALWAYS_TRAVERSE", True, raising=False)
+3 -1
View File
@@ -289,7 +289,7 @@ class Console:
return console.Console(stderr=True) return console.Console(stderr=True)
def table( def table( # noqa: PLR0913
self, self,
data: "TableData", data: "TableData",
headers: Optional["Headers"] = None, headers: Optional["Headers"] = None,
@@ -301,6 +301,7 @@ class Console:
row_styles: Optional[Sequence["Styles"]] = None, row_styles: Optional[Sequence["Styles"]] = None,
borders: Union[bool, str] = False, borders: Union[bool, str] = False,
colalign: Optional[tuple[str, ...]] = None, colalign: Optional[tuple[str, ...]] = None,
preserve_whitespace: bool = False,
) -> None: ) -> None:
from dvc.ui import table as t from dvc.ui import table as t
@@ -329,6 +330,7 @@ class Console:
pager=pager, pager=pager,
force=force, force=force,
colalign=colalign, colalign=colalign,
preserve_whitespace=preserve_whitespace,
) )
def status(self, status: str, **kwargs: Any) -> "Status": def status(self, status: str, **kwargs: Any) -> "Status":
+27 -9
View File
@@ -30,19 +30,37 @@ def plain_table(
pager: bool = False, pager: bool = False,
force: bool = True, force: bool = True,
colalign: Optional[tuple[str, ...]] = None, colalign: Optional[tuple[str, ...]] = None,
preserve_whitespace: bool = False,
) -> None: ) -> None:
import tabulate as tabulate_mod
from funcy import nullcontext from funcy import nullcontext
from packaging.version import Version
from tabulate import tabulate from tabulate import tabulate
text: str = tabulate( # NOTE: tabulate 0.10.0+ supports preserve_whitespace as a kwarg,
data, # while 0.9.0 only respects the global PRESERVE_WHITESPACE.
headers if headers is not None else (), kwargs: dict = {}
tablefmt="github" if markdown else "plain", _orig = tabulate_mod.PRESERVE_WHITESPACE
disable_numparse=True, if preserve_whitespace:
# None will be shown as "" by default, overriding if Version(tabulate_mod.__version__) >= Version("0.10"):
missingval="-", kwargs["preserve_whitespace"] = True
colalign=colalign, else:
) tabulate_mod.PRESERVE_WHITESPACE = True
try:
text: str = tabulate(
data,
headers if headers is not None else (),
tablefmt="github" if markdown else "plain",
disable_numparse=True,
# None will be shown as "" by default, overriding
missingval="-",
colalign=colalign,
**kwargs,
)
finally:
tabulate_mod.PRESERVE_WHITESPACE = _orig
if markdown: if markdown:
# NOTE: md table is incomplete without the trailing newline # NOTE: md table is incomplete without the trailing newline
text += "\n" text += "\n"
+1 -1
View File
@@ -163,7 +163,7 @@ class Updater:
if pkg not in instructions: if pkg not in instructions:
return ( return (
"Find the latest release at " "Find the latest release at "
"[blue]https://github.com/iterative/dvc/releases/latest[/]." "[blue]https://github.com/treeverse/dvc/releases/latest[/]."
) )
instruction = instructions[pkg] instruction = instructions[pkg]
+12 -14
View File
@@ -18,7 +18,7 @@ keywords = [
] ]
license = "Apache-2.0" license = "Apache-2.0"
license-files = ["LICENSE"] license-files = ["LICENSE"]
maintainers = [{ name = "Iterative", email = "[email protected]" }] maintainers = [{ name = "Treeverse", email = "[email protected]" }]
authors = [{ name = "Dmitry Petrov", email = "[email protected]" }] authors = [{ name = "Dmitry Petrov", email = "[email protected]" }]
requires-python = ">=3.9" requires-python = ">=3.9"
classifiers = [ classifiers = [
@@ -29,6 +29,7 @@ classifiers = [
"Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
] ]
dynamic = ["version"] dynamic = ["version"]
@@ -40,14 +41,14 @@ dependencies = [
"distro>=1.3", "distro>=1.3",
"dpath<3,>=2.1.0", "dpath<3,>=2.1.0",
"dulwich", "dulwich",
"dvc-data>=3.16.2,<3.17", "dvc-data>=3.18.2,<3.19.0",
"dvc-http>=2.29.0", "dvc-http>=2.29.0",
"dvc-objects", "dvc-objects",
"dvc-render>=1.0.1,<2", "dvc-render>=1.0.1,<2",
"dvc-studio-client>=0.21,<1", "dvc-studio-client>=0.21,<1",
"dvc-task>=0.3.0,<1", "dvc-task>=0.3.0,<1",
"flatten_dict<1,>=0.4.1", "flatten-dict<1,>=0.4.1",
"flufl.lock>=8.1.0,<9", "flufl.lock>=8.1.0,<10",
"fsspec>=2024.2.0", "fsspec>=2024.2.0",
"funcy>=1.14", "funcy>=1.14",
"grandalf<1,>=0.7", "grandalf<1,>=0.7",
@@ -58,12 +59,12 @@ dependencies = [
"networkx>=2.5", "networkx>=2.5",
"omegaconf", "omegaconf",
"packaging>=19", "packaging>=19",
"pathspec>=0.10.3", "pathspec>=0.10.3,<2",
"platformdirs<5,>=3.1.1", "platformdirs<5,>=3.1.1",
"psutil>=5.8", "psutil>=5.8",
"pydot>=1.2.4", "pydot>=1.2.4",
"pygtrie>=2.3.2", "pygtrie>=2.3.2",
"pyparsing>=2.4.7", "pyparsing>=3.0.0",
"requests>=2.22", "requests>=2.22",
"rich>=12", "rich>=12",
"ruamel.yaml>=0.17.11", "ruamel.yaml>=0.17.11",
@@ -85,7 +86,7 @@ gdrive = ["dvc-gdrive>=3,<4"]
gs = ["dvc-gs>=3.0.2,<4"] gs = ["dvc-gs>=3.0.2,<4"]
hdfs = ["dvc-hdfs>=3,<4"] hdfs = ["dvc-hdfs>=3,<4"]
lint = [ lint = [
"mypy==1.18.2", "mypy==1.19.1",
"pandas-stubs", "pandas-stubs",
"types-colorama", "types-colorama",
"types-psutil", "types-psutil",
@@ -102,7 +103,6 @@ ssh = ["dvc-ssh>=4,<5"]
ssh_gssapi = ["dvc-ssh[gssapi]>=4,<5"] ssh_gssapi = ["dvc-ssh[gssapi]>=4,<5"]
testing = [ testing = [
"pytest-benchmark[histogram]>=5,<6", "pytest-benchmark[histogram]>=5,<6",
"pytest-test-utils",
"uv", "uv",
] ]
tests = [ tests = [
@@ -110,12 +110,11 @@ tests = [
"beautifulsoup4>=4.4", "beautifulsoup4>=4.4",
"dvc-ssh", "dvc-ssh",
"filelock", "filelock",
"pytest>=7,<9", "pytest>=7,<10",
"pytest-rerunfailures<17.0", "pytest-rerunfailures<17.0",
"pytest-cov>=4.1.0", "pytest-cov>=4.1.0",
"pytest-docker>=1,<4", "pytest-docker>=1,<4",
"pytest-mock", "pytest-mock",
"pytest-test-utils",
"pytest-timeout>=2", "pytest-timeout>=2",
"pytest-xdist>=3.2", "pytest-xdist>=3.2",
'pywin32>=225; sys_platform == "win32"', # optional test dependency 'pywin32>=225; sys_platform == "win32"', # optional test dependency
@@ -123,14 +122,14 @@ tests = [
"sqlalchemy>=1,<3", # optional dependency for `import-db` "sqlalchemy>=1,<3", # optional dependency for `import-db`
"pandas>=1", # optional dependency for `import-db` "pandas>=1", # optional dependency for `import-db`
] ]
webdav = ["dvc-webdav>=3,<4"] webdav = ["dvc-webdav>=3.0.1,<4"]
webhdfs = ["dvc-webhdfs>=3.1,<4"] webhdfs = ["dvc-webhdfs>=3.1,<4"]
webhdfs_kerberos = ["dvc-webhdfs[kerberos]>=3.1,<4"] webhdfs_kerberos = ["dvc-webhdfs[kerberos]>=3.1,<4"]
[project.urls] [project.urls]
Documentation = "https://dvc.org/doc" Documentation = "https://dvc.org/doc"
Issues = "https://github.com/iterative/dvc/issues" Issues = "https://github.com/treeverse/dvc/issues"
Source = "https://github.com/iterative/dvc" Source = "https://github.com/treeverse/dvc"
[project.scripts] [project.scripts]
dvc = "dvc.cli:main" dvc = "dvc.cli:main"
@@ -248,7 +247,6 @@ module = [
"shortuuid", "shortuuid",
"shtab", "shtab",
"upath", "upath",
"virtualenv",
"viztracer", "viztracer",
"voluptuous", "voluptuous",
"yappi", "yappi",
-17
View File
@@ -1,17 +0,0 @@
import os
# Increasing fd ulimit for tests
if os.name == "nt":
try:
import win32file
except ImportError:
pass
else:
win32file._setmaxstdio(4096)
else:
import resource
resource.setrlimit(resource.RLIMIT_NOFILE, (4096, 4096))
nproc_soft, nproc_hard = resource.getrlimit(resource.RLIMIT_NPROC)
resource.setrlimit(resource.RLIMIT_NPROC, (nproc_hard, nproc_hard))
-6
View File
@@ -1,6 +0,0 @@
if __name__ == "__main__":
import sys
import pytest
sys.exit(pytest.main(["-v", "-n=logical", "--dist=worksteal", *sys.argv[1:]]))
+1
View File
@@ -194,6 +194,7 @@ def isolate(tmp_path_factory):
monkeypatch.setenv(env_var, os.fspath(path)) monkeypatch.setenv(env_var, os.fspath(path))
else: else:
monkeypatch.setenv("HOME", str(home_dir)) monkeypatch.setenv("HOME", str(home_dir))
monkeypatch.setenv("XDG_CONFIG_HOME", str(home_dir / ".config"))
monkeypatch.setenv("GIT_CONFIG_NOSYSTEM", "1") monkeypatch.setenv("GIT_CONFIG_NOSYSTEM", "1")
contents = b""" contents = b"""
+1 -1
View File
@@ -91,7 +91,7 @@ def test_apply_untracked(tmp_dir, scm, dvc, exp_stage):
def test_apply_unchanged_head(tmp_dir, scm, dvc, exp_stage): def test_apply_unchanged_head(tmp_dir, scm, dvc, exp_stage):
# see https://github.com/iterative/dvc/issues/8764 # see https://github.com/treeverse/dvc/issues/8764
tmp_dir.gen("params.yaml", "foo: 2") tmp_dir.gen("params.yaml", "foo: 2")
scm.add(["dvc.yaml", "dvc.lock", "params.yaml", "metrics.yaml"]) scm.add(["dvc.yaml", "dvc.lock", "params.yaml", "metrics.yaml"])
scm.commit("commit foo=2") scm.commit("commit foo=2")
+1 -1
View File
@@ -53,7 +53,7 @@ def test_new_simple(tmp_dir, scm, dvc, exp_stage, mocker, name, workspace):
if workspace: if workspace:
assert (tmp_dir / "metrics.yaml").read_text().strip() == "foo: 2" assert (tmp_dir / "metrics.yaml").read_text().strip() == "foo: 2"
exp_name = name if name else ref_info.name exp_name = name or ref_info.name
assert dvc.experiments.get_exact_name([exp])[exp] == exp_name assert dvc.experiments.get_exact_name([exp])[exp] == exp_name
assert resolve_rev(scm, exp_name) == exp assert resolve_rev(scm, exp_name) == exp
+1 -1
View File
@@ -26,7 +26,7 @@ def test_celery_logs(tmp_dir, scm, dvc, failed_exp_stage, follow, capsys, test_q
def test_queue_doesnt_remove_untracked_params_file(tmp_dir, dvc, scm): def test_queue_doesnt_remove_untracked_params_file(tmp_dir, dvc, scm):
"""Regression test for https://github.com/iterative/dvc/issues/7842""" """Regression test for https://github.com/treeverse/dvc/issues/7842"""
tmp_dir.gen("params.yaml", "foo: 1") tmp_dir.gen("params.yaml", "foo: 1")
stage = dvc.run(cmd="echo ${foo}", params=["foo"], name="echo-foo") stage = dvc.run(cmd="echo ${foo}", params=["foo"], name="echo-foo")
scm.add(["dvc.yaml", "dvc.lock", ".gitignore"]) scm.add(["dvc.yaml", "dvc.lock", ".gitignore"])
+2 -2
View File
@@ -28,7 +28,7 @@ def test_exp_save(tmp_dir, dvc, scm, name):
assert ref_info assert ref_info
assert ref_info.baseline_sha == baseline assert ref_info.baseline_sha == baseline
exp_name = name if name else ref_info.name exp_name = name or ref_info.name
assert dvc.experiments.get_exact_name([exp])[exp] == exp_name assert dvc.experiments.get_exact_name([exp])[exp] == exp_name
assert resolve_rev(scm, exp_name) == exp assert resolve_rev(scm, exp_name) == exp
@@ -105,7 +105,7 @@ def test_exp_save_include_untracked(tmp_dir, dvc, scm):
def test_exp_save_include_untracked_warning(tmp_dir, dvc, scm, mocker): def test_exp_save_include_untracked_warning(tmp_dir, dvc, scm, mocker):
"""Regression test for https://github.com/iterative/dvc/issues/9061""" """Regression test for https://github.com/treeverse/dvc/issues/9061"""
setup_stage(tmp_dir, dvc, scm) setup_stage(tmp_dir, dvc, scm)
new_dir = tmp_dir / "new_dir" new_dir = tmp_dir / "new_dir"
+79 -4
View File
@@ -26,8 +26,9 @@ def test_modify_params(params_repo, dvc, changes, expected):
("conf", "bar"), ("conf", "bar"),
], ],
) )
@pytest.mark.parametrize("no_hydra", [True, False])
def test_hydra_compose_and_dump( def test_hydra_compose_and_dump(
tmp_dir, params_repo, dvc, hydra_enabled, config_dir, config_name tmp_dir, params_repo, dvc, hydra_enabled, config_dir, config_name, no_hydra
): ):
hydra_setup( hydra_setup(
tmp_dir, tmp_dir,
@@ -50,14 +51,14 @@ def test_hydra_compose_and_dump(
if config_name is not None: if config_name is not None:
conf["hydra"]["config_name"] = config_name conf["hydra"]["config_name"] = config_name
dvc.experiments.run() dvc.experiments.run(no_hydra=no_hydra)
if hydra_enabled: if hydra_enabled and not no_hydra:
assert (tmp_dir / "params.yaml").parse() == { assert (tmp_dir / "params.yaml").parse() == {
"db": {"driver": "mysql", "user": "omry", "pass": "secret"}, "db": {"driver": "mysql", "user": "omry", "pass": "secret"},
} }
dvc.experiments.run(params=["db=postgresql"]) dvc.experiments.run(params=["db=postgresql"], no_hydra=no_hydra)
assert (tmp_dir / "params.yaml").parse() == { assert (tmp_dir / "params.yaml").parse() == {
"db": { "db": {
"driver": "postgresql", "driver": "postgresql",
@@ -117,6 +118,7 @@ def test_hydra_sweep(
targets=None, targets=None,
copy_paths=None, copy_paths=None,
message=None, message=None,
no_hydra=False,
) )
@@ -137,3 +139,76 @@ def test_hydra_sweep_prefix_name(tmp_dir, params_repo, dvc):
exp_names = [entry.name for entry in dvc.experiments.celery_queue.iter_queued()] exp_names = [entry.name for entry in dvc.experiments.celery_queue.iter_queued()]
for name, expected in zip(exp_names, expected_names): for name, expected in zip(exp_names, expected_names):
assert name == expected assert name == expected
def test_mixing_no_hydra_and_params_flags(tmp_dir, params_repo, dvc):
# Passing no_hydra should not prevent user from
# using --set-param on unmodified params.yaml
hydra_setup(
tmp_dir,
config_dir="conf",
config_name="config",
)
with dvc.config.edit() as conf:
conf["hydra"]["enabled"] = True
conf["hydra"]["config_dir"] = "conf"
conf["hydra"]["config_name"] = "config"
dvc.experiments.run(no_hydra=True, params=["goo.bag=10.0"])
assert (tmp_dir / "params.yaml").parse() == {
"foo": [{"bar": 1}, {"baz": 2}],
"goo": {"bag": 10.0},
"lorem": False,
}
@pytest.mark.parametrize(
"hydra_enabled,overrides,expected",
[
(
True,
["db=mysql,postgresql"],
[
{"params.yaml": ["db=mysql"]},
{"params.yaml": ["db=postgresql"]},
],
),
(
False,
["foo=bar,baz"],
[{"params.yaml": ["foo=bar"]}, {"params.yaml": ["foo=baz"]}],
),
(
False,
[],
[{}],
),
],
)
@pytest.mark.parametrize("no_hydra", [True, False])
def test_mixing_no_hydra_and_sweeps(
tmp_dir, params_repo, dvc, mocker, hydra_enabled, overrides, expected, no_hydra
):
# Passing no_hydra should not prevent user from
# queuing sweeps with --set-param and --queue
patched = mocker.patch.object(dvc.experiments, "queue_one")
if hydra_enabled:
hydra_setup(tmp_dir, config_dir="conf", config_name="config")
with dvc.config.edit() as conf:
conf["hydra"]["enabled"] = True
dvc.experiments.run(params=overrides, queue=True, no_hydra=no_hydra)
assert patched.call_count == len(expected)
for e in expected:
patched.assert_any_call(
mocker.ANY,
params=e,
targets=None,
copy_paths=None,
message=None,
no_hydra=no_hydra,
)
+3 -2
View File
@@ -4,6 +4,7 @@ from os.path import join
import pytest import pytest
from dvc.cli import main from dvc.cli import main
from dvc.testing import matchers as M
from dvc.utils import relpath from dvc.utils import relpath
from dvc.utils.serialize import JSONFileCorruptedError from dvc.utils.serialize import JSONFileCorruptedError
@@ -95,7 +96,7 @@ def test_metrics_diff_json_unchanged(tmp_dir, scm, dvc, run_copy_metrics):
assert dvc.metrics.diff(a_rev="HEAD~2") == {} assert dvc.metrics.diff(a_rev="HEAD~2") == {}
def test_metrics_diff_broken_json(M, tmp_dir, scm, dvc, run_copy_metrics): def test_metrics_diff_broken_json(tmp_dir, scm, dvc, run_copy_metrics):
metrics = {"a": {"b": {"c": 1, "d": 1, "e": "3"}}} metrics = {"a": {"b": {"c": 1, "d": 1, "e": "3"}}}
(tmp_dir / "m_temp.json").dump(metrics) (tmp_dir / "m_temp.json").dump(metrics)
run_copy_metrics( run_copy_metrics(
@@ -143,7 +144,7 @@ def test_metrics_diff_new_metric(tmp_dir, scm, dvc, run_copy_metrics):
} }
def test_metrics_diff_deleted_metric(M, tmp_dir, scm, dvc, run_copy_metrics): def test_metrics_diff_deleted_metric(tmp_dir, scm, dvc, run_copy_metrics):
metrics = {"a": {"b": {"c": 1, "d": 1, "e": "3"}}} metrics = {"a": {"b": {"c": 1, "d": 1, "e": "3"}}}
(tmp_dir / "m_temp.json").dump(metrics) (tmp_dir / "m_temp.json").dump(metrics)
run_copy_metrics( run_copy_metrics(
+3 -2
View File
@@ -11,6 +11,7 @@ from dvc.dvcfile import PROJECT_FILE
from dvc.exceptions import OverlappingOutputPathsError from dvc.exceptions import OverlappingOutputPathsError
from dvc.repo import Repo from dvc.repo import Repo
from dvc.repo.metrics.show import FileResult, Result from dvc.repo.metrics.show import FileResult, Result
from dvc.testing import matchers as M
from dvc.utils.fs import remove from dvc.utils.fs import remove
from dvc.utils.serialize import JSONFileCorruptedError from dvc.utils.serialize import JSONFileCorruptedError
from dvc_data.index import DataIndexDirError from dvc_data.index import DataIndexDirError
@@ -246,7 +247,7 @@ def test_show_malformed_metric(tmp_dir, scm, dvc, caplog):
) )
def test_metrics_show_no_target(M, tmp_dir, dvc, capsys): def test_metrics_show_no_target(tmp_dir, dvc, capsys):
assert dvc.metrics.show(targets=["metrics.json"]) == { assert dvc.metrics.show(targets=["metrics.json"]) == {
"": {"data": {"metrics.json": {"error": M.instance_of(FileNotFoundError)}}} "": {"data": {"metrics.json": {"error": M.instance_of(FileNotFoundError)}}}
} }
@@ -356,7 +357,7 @@ def test_top_level_parametrized(tmp_dir, dvc):
} }
def test_metric_in_a_tracked_directory_with_missing_dir_file(M, tmp_dir, dvc): def test_metric_in_a_tracked_directory_with_missing_dir_file(tmp_dir, dvc):
tmp_dir.dvc_gen({"dir": {"file": "2"}}) tmp_dir.dvc_gen({"dir": {"file": "2"}})
(tmp_dir / "dvc.yaml").dump({"metrics": [join("dir", "file")]}) (tmp_dir / "dvc.yaml").dump({"metrics": [join("dir", "file")]})
shutil.rmtree(tmp_dir / "dir") # remove from workspace shutil.rmtree(tmp_dir / "dir") # remove from workspace
+2 -1
View File
@@ -2,6 +2,7 @@ from os.path import join
import pytest import pytest
from dvc.testing import matchers as M
from dvc.utils import relpath from dvc.utils import relpath
@@ -54,7 +55,7 @@ def test_diff_new(tmp_dir, scm, dvc):
} }
def test_diff_deleted(M, tmp_dir, scm, dvc): def test_diff_deleted(tmp_dir, scm, dvc):
tmp_dir.gen("params.yaml", "foo: bar") tmp_dir.gen("params.yaml", "foo: bar")
dvc.run(cmd="echo params.yaml", params=["foo"], name="echo-params") dvc.run(cmd="echo params.yaml", params=["foo"], name="echo-params")
scm.add(["params.yaml", "Dvcfile"]) scm.add(["params.yaml", "Dvcfile"])
+2 -1
View File
@@ -6,6 +6,7 @@ import pytest
from dvc.dvcfile import PROJECT_FILE from dvc.dvcfile import PROJECT_FILE
from dvc.repo import Repo from dvc.repo import Repo
from dvc.repo.metrics.show import FileResult, Result from dvc.repo.metrics.show import FileResult, Result
from dvc.testing import matchers as M
from dvc_data.index import DataIndexDirError from dvc_data.index import DataIndexDirError
@@ -197,7 +198,7 @@ def test_top_level_parametrized(tmp_dir, dvc):
} }
def test_param_in_a_tracked_directory_with_missing_dir_file(M, tmp_dir, dvc): def test_param_in_a_tracked_directory_with_missing_dir_file(tmp_dir, dvc):
tmp_dir.dvc_gen({"dir": {"file": "2"}}) tmp_dir.dvc_gen({"dir": {"file": "2"}})
(tmp_dir / "dvc.yaml").dump({"params": [join("dir", "file")]}) (tmp_dir / "dvc.yaml").dump({"params": [join("dir", "file")]})
shutil.rmtree(tmp_dir / "dir") # remove from workspace shutil.rmtree(tmp_dir / "dir") # remove from workspace
+3 -2
View File
@@ -15,6 +15,7 @@ from dvc.output import Output
from dvc.stage import PipelineStage, Stage from dvc.stage import PipelineStage, Stage
from dvc.stage.cache import RunCacheNotSupported from dvc.stage.cache import RunCacheNotSupported
from dvc.stage.exceptions import StageFileDoesNotExistError, StageNotFound from dvc.stage.exceptions import StageFileDoesNotExistError, StageNotFound
from dvc.testing import matchers as M
from dvc.utils.fs import remove from dvc.utils.fs import remove
from dvc.utils.serialize import modify_yaml from dvc.utils.serialize import modify_yaml
from dvc_data.hashfile.hash import file_md5 from dvc_data.hashfile.hash import file_md5
@@ -55,7 +56,7 @@ def test_repro_frozen(tmp_dir, dvc, run_copy):
assert stages == [data_stage, stage0] assert stages == [data_stage, stage0]
def test_downstream(M, tmp_dir, dvc): def test_downstream(tmp_dir, dvc):
# The dependency graph should look like this: # The dependency graph should look like this:
# #
# E # E
@@ -148,7 +149,7 @@ def test_downstream(M, tmp_dir, dvc):
assert [stage.name for stage in evaluation] == ["B-gen", "D-gen", "E-gen"] assert [stage.name for stage in evaluation] == ["B-gen", "D-gen", "E-gen"]
# B, C should be run (in any order) before D # B, C should be run (in any order) before D
# See https://github.com/iterative/dvc/issues/3602 # See https://github.com/treeverse/dvc/issues/3602
evaluation = dvc.reproduce(PROJECT_FILE + ":A-gen", downstream=True, force=True) evaluation = dvc.reproduce(PROJECT_FILE + ":A-gen", downstream=True, force=True)
assert len(evaluation) == 5 assert len(evaluation) == 5
+1 -1
View File
@@ -36,7 +36,7 @@ def test_repro_allow_missing_and_pull(tmp_dir, dvc, mocker, local_remote):
def test_repro_allow_missing_upstream_stage_modified( def test_repro_allow_missing_upstream_stage_modified(
tmp_dir, dvc, mocker, local_remote tmp_dir, dvc, mocker, local_remote
): ):
"""https://github.com/iterative/dvc/issues/9530""" """https://github.com/treeverse/dvc/issues/9530"""
tmp_dir.gen("params.yaml", "param: 1") tmp_dir.gen("params.yaml", "param: 1")
dvc.stage.add( dvc.stage.add(
name="create-foo", cmd="echo ${param} > foo", params=["param"], outs=["foo"] name="create-foo", cmd="echo ${param} > foo", params=["param"], outs=["foo"]
+1 -1
View File
@@ -723,7 +723,7 @@ def test_add_symlink_file(tmp_dir, dvc):
).is_symlink() ).is_symlink()
# Test that subsequent add succeeds # Test that subsequent add succeeds
# See https://github.com/iterative/dvc/issues/4654 # See https://github.com/treeverse/dvc/issues/4654
dvc.add(os.path.join("dir", "foo")) dvc.add(os.path.join("dir", "foo"))
+6 -6
View File
@@ -88,8 +88,8 @@ def test_config_set_in_non_dvc_repo(tmp_dir, caplog):
[ [
(["core.analytics"], 0, "False"), (["core.analytics"], 0, "False"),
(["core.remote"], 0, "myremote"), (["core.remote"], 0, "myremote"),
(["remote.myremote.profile"], 0, "iterative"), (["remote.myremote.profile"], 0, "treeverse"),
(["remote.myremote.profile", "--local"], 0, "iterative"), (["remote.myremote.profile", "--local"], 0, "treeverse"),
( (
["remote.myremote.profile", "--project"], ["remote.myremote.profile", "--project"],
251, 251,
@@ -122,7 +122,7 @@ def test_config_get(tmp_dir, dvc, capsys, caplog, args, ret, msg):
textwrap.dedent( textwrap.dedent(
"""\ """\
['remote "myremote"'] ['remote "myremote"']
profile = iterative profile = treeverse
['remote "other"'] ['remote "other"']
url = gs://bucket/path url = gs://bucket/path
""" """
@@ -167,7 +167,7 @@ def test_config_list(tmp_dir, dvc, capsys):
textwrap.dedent( textwrap.dedent(
"""\ """\
['remote "myremote"'] ['remote "myremote"']
profile = iterative profile = treeverse
access_key_id = abcde access_key_id = abcde
secret_access_key = 123456 secret_access_key = 123456
['remote "other"'] ['remote "other"']
@@ -181,7 +181,7 @@ def test_config_list(tmp_dir, dvc, capsys):
out, _ = capsys.readouterr() out, _ = capsys.readouterr()
assert "remote.myremote.url=s3://bucket/path" in out assert "remote.myremote.url=s3://bucket/path" in out
assert "remote.myremote.region=us-east-2" in out assert "remote.myremote.region=us-east-2" in out
assert "remote.myremote.profile=iterative" in out assert "remote.myremote.profile=treeverse" in out
assert "remote.myremote.access_key_id=abcde" in out assert "remote.myremote.access_key_id=abcde" in out
assert "remote.myremote.secret_access_key=123456" in out assert "remote.myremote.secret_access_key=123456" in out
assert "remote.other.url=gs://bucket/path" in out assert "remote.other.url=gs://bucket/path" in out
@@ -242,7 +242,7 @@ def test_merging_two_levels(dvc):
def test_config_loads_without_error_for_non_dvc_repo(tmp_dir): def test_config_loads_without_error_for_non_dvc_repo(tmp_dir):
# regression testing for https://github.com/iterative/dvc/issues/3328 # regression testing for https://github.com/treeverse/dvc/issues/3328
Config.from_cwd(validate=True) Config.from_cwd(validate=True)
+2 -2
View File
@@ -730,12 +730,12 @@ def test_pull_allow_missing(tmp_dir, dvc, local_remote):
def test_pull_granular_excluding_import_that_cannot_be_pulled( def test_pull_granular_excluding_import_that_cannot_be_pulled(
tmp_dir, dvc, local_remote, mocker tmp_dir, dvc, local_remote, mocker
): ):
"""Regression test for https://github.com/iterative/dvc/issues/10309.""" """Regression test for https://github.com/treeverse/dvc/issues/10309."""
mocker.patch("dvc.fs.dvc._DVCFileSystem", side_effect=CloneError("SCM error")) mocker.patch("dvc.fs.dvc._DVCFileSystem", side_effect=CloneError("SCM error"))
(stage,) = tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}) (stage,) = tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}})
imp_stage = dvc.imp( imp_stage = dvc.imp(
"https://user:[email protected]/iterative/dvc.git", "https://user:[email protected]/treeverse/dvc.git",
"dir", "dir",
out="new_dir", out="new_dir",
rev="HEAD", rev="HEAD",
+105 -48
View File
@@ -6,10 +6,10 @@ from os.path import join
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
import pytest import pytest
from pytest_test_utils import matchers as m
from dvc.repo import Repo from dvc.repo import Repo
from dvc.repo.data import _transform_git_paths_to_dvc, posixpath_to_os_path from dvc.repo.data import _transform_git_paths_to_dvc, posixpath_to_os_path
from dvc.testing import matchers as M
from dvc.testing.tmp_dir import TmpDir, make_subrepo from dvc.testing.tmp_dir import TmpDir, make_subrepo
from dvc.utils.fs import remove from dvc.utils.fs import remove
@@ -60,7 +60,7 @@ def test_git_to_dvc_path_wdir_transformation(tmp_dir, scm, path):
] ]
def test_file(M, tmp_dir, dvc, scm): def test_file(tmp_dir, dvc, scm):
tmp_dir.dvc_gen("foo", "foo", commit="add foo") tmp_dir.dvc_gen("foo", "foo", commit="add foo")
tmp_dir.dvc_gen("foo", "foobar") tmp_dir.dvc_gen("foo", "foobar")
remove(tmp_dir / "foo") remove(tmp_dir / "foo")
@@ -75,7 +75,7 @@ def test_file(M, tmp_dir, dvc, scm):
assert dvc.data_status(granular=True) == expected assert dvc.data_status(granular=True) == expected
def test_directory(M, tmp_dir, dvc, scm): def test_directory(tmp_dir, dvc, scm):
tmp_dir.dvc_gen({"dir": {"foo": "foo"}}, commit="add dir") tmp_dir.dvc_gen({"dir": {"foo": "foo"}}, commit="add dir")
tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar", "foobar": "foobar"}}) tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar", "foobar": "foobar"}})
remove(tmp_dir / "dir") remove(tmp_dir / "dir")
@@ -107,7 +107,7 @@ def test_directory(M, tmp_dir, dvc, scm):
} }
def test_tracked_directory_deep(M, tmp_dir, dvc, scm): def test_tracked_directory_deep(tmp_dir, dvc, scm):
"""Test for a directory not in cwd, but nested inside other directories.""" """Test for a directory not in cwd, but nested inside other directories."""
(tmp_dir / "sub").gen({"dir": {"foo": "foo"}}) (tmp_dir / "sub").gen({"dir": {"foo": "foo"}})
dvc.add(fspath(tmp_dir / "sub" / "dir")) dvc.add(fspath(tmp_dir / "sub" / "dir"))
@@ -140,7 +140,7 @@ def test_tracked_directory_deep(M, tmp_dir, dvc, scm):
@pytest.mark.parametrize("git_repo_state", ["unborn", "committed"]) @pytest.mark.parametrize("git_repo_state", ["unborn", "committed"])
@pytest.mark.parametrize("subdir", [True, False]) @pytest.mark.parametrize("subdir", [True, False])
def test_new_dvc_repo(M, tmp_dir, scm, subdir, git_repo_state): def test_new_dvc_repo(tmp_dir, scm, subdir, git_repo_state):
if git_repo_state == "committed": if git_repo_state == "committed":
tmp_dir.scm_gen("test", "test", commit="init") tmp_dir.scm_gen("test", "test", commit="init")
@@ -167,7 +167,7 @@ def test_noscm_repo(tmp_dir, dvc):
assert dvc.data_status() == {**EMPTY_STATUS, "unchanged": ["foo"]} assert dvc.data_status() == {**EMPTY_STATUS, "unchanged": ["foo"]}
def test_unchanged(M, tmp_dir, dvc, scm): def test_unchanged(tmp_dir, dvc, scm):
tmp_dir.dvc_gen({"dir": {"foo": "foo"}}, commit="add dir") tmp_dir.dvc_gen({"dir": {"foo": "foo"}}, commit="add dir")
tmp_dir.dvc_gen("bar", "bar", commit="add foo") tmp_dir.dvc_gen("bar", "bar", commit="add foo")
@@ -195,7 +195,7 @@ def test_skip_uncached_pipeline_outputs(tmp_dir, dvc, run_copy_metrics):
assert dvc.data_status(granular=True, untracked_files="all") == EMPTY_STATUS assert dvc.data_status(granular=True, untracked_files="all") == EMPTY_STATUS
def test_outs_with_no_hashes(M, tmp_dir, dvc, scm): def test_outs_with_no_hashes(tmp_dir, dvc, scm):
dvc.stage.add(single_stage=True, outs=["bar"]) dvc.stage.add(single_stage=True, outs=["bar"])
dvc.stage.add(deps=["bar"], outs=["foo"], name="copy", cmd="cp foo bar") dvc.stage.add(deps=["bar"], outs=["foo"], name="copy", cmd="cp foo bar")
@@ -204,7 +204,7 @@ def test_outs_with_no_hashes(M, tmp_dir, dvc, scm):
assert dvc.data_status(granular=True) == expected_output assert dvc.data_status(granular=True) == expected_output
def test_outs_with_no_hashes_and_with_uncommitted_files(M, tmp_dir, dvc, scm): def test_outs_with_no_hashes_and_with_uncommitted_files(tmp_dir, dvc, scm):
tmp_dir.gen({"bar": "bar", "foo": "foo"}) tmp_dir.gen({"bar": "bar", "foo": "foo"})
dvc.stage.add(single_stage=True, outs=["bar"]) dvc.stage.add(single_stage=True, outs=["bar"])
dvc.stage.add(deps=["bar"], outs=["foo"], name="copy", cmd="cp foo bar") dvc.stage.add(deps=["bar"], outs=["foo"], name="copy", cmd="cp foo bar")
@@ -218,7 +218,7 @@ def test_outs_with_no_hashes_and_with_uncommitted_files(M, tmp_dir, dvc, scm):
assert dvc.data_status(granular=True) == expected_output assert dvc.data_status(granular=True) == expected_output
def test_subdir(M, tmp_dir, scm): def test_subdir(tmp_dir, scm):
subrepo = tmp_dir / "sub" subrepo = tmp_dir / "sub"
make_subrepo(subrepo, scm) make_subrepo(subrepo, scm)
@@ -236,7 +236,7 @@ def test_subdir(M, tmp_dir, scm):
} }
def test_untracked_newly_added_files(M, tmp_dir, dvc, scm): def test_untracked_newly_added_files(tmp_dir, dvc, scm):
tmp_dir.gen({"dir": {"foo": "foo", "bar": "bar"}}) tmp_dir.gen({"dir": {"foo": "foo", "bar": "bar"}})
tmp_dir.gen("foobar", "foobar") tmp_dir.gen("foobar", "foobar")
@@ -249,7 +249,7 @@ def test_untracked_newly_added_files(M, tmp_dir, dvc, scm):
assert dvc.data_status(granular=True, untracked_files="all") == expected assert dvc.data_status(granular=True, untracked_files="all") == expected
def test_missing_cache_workspace_exists(M, tmp_dir, dvc, scm): def test_missing_cache_workspace_exists(tmp_dir, dvc, scm):
tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}) tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}})
tmp_dir.dvc_gen("foobar", "foobar") tmp_dir.dvc_gen("foobar", "foobar")
remove(dvc.cache.repo.path) remove(dvc.cache.repo.path)
@@ -272,7 +272,7 @@ def test_missing_cache_workspace_exists(M, tmp_dir, dvc, scm):
} }
def test_missing_cache_missing_workspace(M, tmp_dir, dvc, scm): def test_missing_cache_missing_workspace(tmp_dir, dvc, scm):
tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}) tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}})
tmp_dir.dvc_gen("foobar", "foobar") tmp_dir.dvc_gen("foobar", "foobar")
for path in [dvc.cache.repo.path, "dir", "foobar"]: for path in [dvc.cache.repo.path, "dir", "foobar"]:
@@ -297,7 +297,7 @@ def test_missing_cache_missing_workspace(M, tmp_dir, dvc, scm):
} }
def test_git_committed_missing_cache_workspace_exists(M, tmp_dir, dvc, scm): def test_git_committed_missing_cache_workspace_exists(tmp_dir, dvc, scm):
tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}, commit="add dir") tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}, commit="add dir")
tmp_dir.dvc_gen("foobar", "foobar", commit="add foobar") tmp_dir.dvc_gen("foobar", "foobar", commit="add foobar")
remove(dvc.cache.local.path) remove(dvc.cache.local.path)
@@ -317,7 +317,7 @@ def test_git_committed_missing_cache_workspace_exists(M, tmp_dir, dvc, scm):
} }
def test_git_committed_missing_cache_missing_workspace(M, tmp_dir, dvc, scm): def test_git_committed_missing_cache_missing_workspace(tmp_dir, dvc, scm):
tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}, commit="add dir") tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}, commit="add dir")
tmp_dir.dvc_gen("foobar", "foobar", commit="add foobar") tmp_dir.dvc_gen("foobar", "foobar", commit="add foobar")
for path in [dvc.cache.repo.path, "dir", "foobar"]: for path in [dvc.cache.repo.path, "dir", "foobar"]:
@@ -338,7 +338,7 @@ def test_git_committed_missing_cache_missing_workspace(M, tmp_dir, dvc, scm):
} }
def test_partial_missing_cache(M, tmp_dir, dvc, scm): def test_partial_missing_cache(tmp_dir, dvc, scm):
tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}) tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}})
# remove "foo" from cache # remove "foo" from cache
@@ -362,7 +362,7 @@ def test_partial_missing_cache(M, tmp_dir, dvc, scm):
} }
def test_missing_dir_object_from_head(M, tmp_dir, dvc, scm): def test_missing_dir_object_from_head(tmp_dir, dvc, scm):
(stage,) = tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}, commit="add dir") (stage,) = tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}, commit="add dir")
remove("dir") remove("dir")
tmp_dir.dvc_gen({"dir": {"foobar": "foobar"}}) tmp_dir.dvc_gen({"dir": {"foobar": "foobar"}})
@@ -384,7 +384,7 @@ def test_missing_dir_object_from_head(M, tmp_dir, dvc, scm):
} }
def test_missing_dir_object_from_index(M, tmp_dir, dvc, scm): def test_missing_dir_object_from_index(tmp_dir, dvc, scm):
tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}, commit="add dir") tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}, commit="add dir")
remove("dir") remove("dir")
(stage,) = tmp_dir.dvc_gen({"dir": {"foobar": "foobar"}}) (stage,) = tmp_dir.dvc_gen({"dir": {"foobar": "foobar"}})
@@ -406,7 +406,7 @@ def test_missing_dir_object_from_index(M, tmp_dir, dvc, scm):
} }
def test_remote_check(M, tmp_dir, dvc, scm, make_remote): def test_remote_check(tmp_dir, dvc, scm, make_remote):
make_remote("default", default=True) make_remote("default", default=True)
make_remote("myremote", default=False) make_remote("myremote", default=False)
@@ -442,7 +442,7 @@ def test_remote_check(M, tmp_dir, dvc, scm, make_remote):
assert dvc.data_status(granular=True, **opts) == expected_g assert dvc.data_status(granular=True, **opts) == expected_g
def test_missing_remote_cache(M, tmp_dir, dvc, scm, local_remote): def test_missing_remote_cache(tmp_dir, dvc, scm, local_remote):
tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}) tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}})
tmp_dir.dvc_gen("foobar", "foobar") tmp_dir.dvc_gen("foobar", "foobar")
@@ -485,7 +485,7 @@ def test_missing_remote_cache(M, tmp_dir, dvc, scm, local_remote):
def test_not_in_remote_respects_not_pushable( def test_not_in_remote_respects_not_pushable(
M: type["m.Matcher"], tmp_dir: TmpDir, dvc: Repo, scm, mocker, local_remote tmp_dir: TmpDir, dvc: Repo, scm, mocker, local_remote
): ):
stages: list[Stage] = tmp_dir.dvc_gen({"foo": "foo", "dir": {"foobar": "foobar"}}) stages: list[Stage] = tmp_dir.dvc_gen({"foo": "foo", "dir": {"foobar": "foobar"}})
# Make foo not pushable # Make foo not pushable
@@ -529,7 +529,7 @@ def test_not_in_remote_respects_not_pushable(
) )
def test_root_from_dir_to_file(M, tmp_dir, dvc, scm): def test_root_from_dir_to_file(tmp_dir, dvc, scm):
tmp_dir.dvc_gen({"data": {"foo": "foo", "bar": "bar"}}) tmp_dir.dvc_gen({"data": {"foo": "foo", "bar": "bar"}})
remove("data") remove("data")
tmp_dir.gen("data", "file") tmp_dir.gen("data", "file")
@@ -555,7 +555,7 @@ def test_root_from_dir_to_file(M, tmp_dir, dvc, scm):
} }
def test_root_from_file_to_dir(M, tmp_dir, dvc, scm): def test_root_from_file_to_dir(tmp_dir, dvc, scm):
tmp_dir.dvc_gen("data", "file") tmp_dir.dvc_gen("data", "file")
remove("data") remove("data")
tmp_dir.gen({"data": {"foo": "foo", "bar": "bar"}}) tmp_dir.gen({"data": {"foo": "foo", "bar": "bar"}})
@@ -577,8 +577,8 @@ def test_root_from_file_to_dir(M, tmp_dir, dvc, scm):
} }
def test_empty_dir(tmp_dir, scm, dvc, M): def test_empty_dir(tmp_dir, scm, dvc):
# regression testing for https://github.com/iterative/dvc/issues/8958 # regression testing for https://github.com/treeverse/dvc/issues/8958
tmp_dir.dvc_gen({"data": {"foo": "foo"}}) tmp_dir.dvc_gen({"data": {"foo": "foo"}})
remove("data") remove("data")
@@ -592,7 +592,7 @@ def test_empty_dir(tmp_dir, scm, dvc, M):
} }
def test_untracked_files_filter_targets(M, tmp_dir, scm, dvc): def test_untracked_files_filter_targets(tmp_dir, scm, dvc):
tmp_dir.gen( tmp_dir.gen(
{"spam": "spam", "ham": "ham", "dir": {"eggs": "eggs", "bacon": "bacon"}} {"spam": "spam", "ham": "ham", "dir": {"eggs": "eggs", "bacon": "bacon"}}
) )
@@ -644,13 +644,13 @@ def param(*values):
param( param(
["foo", "foobar"], ["foo", "foobar"],
{ {
"committed": {"added": m.unordered("foo", "foobar")}, "committed": {"added": M.unordered("foo", "foobar")},
"uncommitted": {"deleted": ["foo"]}, "uncommitted": {"deleted": ["foo"]},
}, },
), ),
], ],
) )
def test_filter_targets_files_after_dvc_commit(M, tmp_dir, dvc, scm, targets, expected): def test_filter_targets_files_after_dvc_commit(tmp_dir, dvc, scm, targets, expected):
tmp_dir.dvc_gen({"foo": "foo", "bar": "bar", "foobar": "foobar"}) tmp_dir.dvc_gen({"foo": "foo", "bar": "bar", "foobar": "foobar"})
(tmp_dir / "foo").unlink() # deleted (tmp_dir / "foo").unlink() # deleted
tmp_dir.gen({"bar": "bar modified", "baz": "baz new"}) tmp_dir.gen({"bar": "bar modified", "baz": "baz new"})
@@ -677,7 +677,7 @@ def test_filter_targets_files_after_dvc_commit(M, tmp_dir, dvc, scm, targets, ex
), ),
], ],
) )
def test_filter_targets_after_git_commit(M, tmp_dir, dvc, scm, targets, expected): def test_filter_targets_after_git_commit(tmp_dir, dvc, scm, targets, expected):
tmp_dir.dvc_gen( tmp_dir.dvc_gen(
{"foo": "foo", "bar": "bar", "foobar": "foobar", "baz": "baz"}, {"foo": "foo", "bar": "bar", "foobar": "foobar", "baz": "baz"},
commit="add files", commit="add files",
@@ -718,7 +718,7 @@ def with_aliases(values, aliases):
}, },
{ {
"committed": { "committed": {
"added": m.unordered( "added": M.unordered(
join("dir", ""), join("dir", ""),
join("dir", "foo"), join("dir", "foo"),
join("dir", "sub", "bar"), join("dir", "sub", "bar"),
@@ -779,7 +779,7 @@ def with_aliases(values, aliases):
{}, {},
{ {
"committed": { "committed": {
"added": m.unordered(join("dir", "foo"), join("dir", "foobar")) "added": M.unordered(join("dir", "foo"), join("dir", "foobar"))
}, },
"uncommitted": {"deleted": [join("dir", "foo")]}, "uncommitted": {"deleted": [join("dir", "foo")]},
}, },
@@ -794,7 +794,7 @@ def with_aliases(values, aliases):
), ),
) )
def test_filter_targets_inside_directory_after_dvc_commit( def test_filter_targets_inside_directory_after_dvc_commit(
M, tmp_dir, dvc, scm, targets, expected_ng, expected_g tmp_dir, dvc, scm, targets, expected_ng, expected_g
): ):
tmp_dir.dvc_gen({"dir": {"foo": "foo", "sub": {"bar": "bar"}, "foobar": "foobar"}}) tmp_dir.dvc_gen({"dir": {"foo": "foo", "sub": {"bar": "bar"}, "foobar": "foobar"}})
(tmp_dir / "dir" / "foo").unlink() # deleted (tmp_dir / "dir" / "foo").unlink() # deleted
@@ -827,7 +827,7 @@ def test_filter_targets_inside_directory_after_dvc_commit(
"unchanged": [join("dir", "foobar")], "unchanged": [join("dir", "foobar")],
"committed": { "committed": {
"added": [join("dir", "baz")], "added": [join("dir", "baz")],
"modified": m.unordered( "modified": M.unordered(
join("dir", ""), join("dir", "sub", "bar") join("dir", ""), join("dir", "sub", "bar")
), ),
"deleted": [join("dir", "foo")], "deleted": [join("dir", "foo")],
@@ -861,7 +861,7 @@ def test_filter_targets_inside_directory_after_dvc_commit(
), ),
) )
def test_filter_targets_inside_directory_after_git_commit( def test_filter_targets_inside_directory_after_git_commit(
M, tmp_dir, dvc, scm, targets, expected_ng, expected_g tmp_dir, dvc, scm, targets, expected_ng, expected_g
): ):
tmp_dir.dvc_gen( tmp_dir.dvc_gen(
{"dir": {"foo": "foo", "sub": {"bar": "bar"}, "foobar": "foobar"}}, {"dir": {"foo": "foo", "sub": {"bar": "bar"}, "foobar": "foobar"}},
@@ -893,22 +893,22 @@ def test_filter_targets_inside_directory_after_git_commit(
param( param(
(join("dir", "bar"), "foo"), (join("dir", "bar"), "foo"),
["foo"], ["foo"],
m.unordered(join("dir", "bar"), "foo"), M.unordered(join("dir", "bar"), "foo"),
), ),
param( param(
(join("dir", "bar"), "dir"), (join("dir", "bar"), "dir"),
[join("dir", "")], [join("dir", "")],
m.unordered(join("dir", ""), join("dir", "bar")), M.unordered(join("dir", ""), join("dir", "bar")),
), ),
param( param(
("dir", "foo"), ("dir", "foo"),
m.unordered(join("dir", ""), "foo"), M.unordered(join("dir", ""), "foo"),
m.unordered(join("dir", ""), join("dir", "bar"), "foo"), M.unordered(join("dir", ""), join("dir", "bar"), "foo"),
), ),
], ],
) )
def test_filter_targets_not_in_cache( def test_filter_targets_not_in_cache(
M, local_remote, tmp_dir, scm, dvc, to_check, targets, non_granular, granular local_remote, tmp_dir, scm, dvc, to_check, targets, non_granular, granular
): ):
tmp_dir.dvc_gen({"foo": "foo", "dir": {"bar": "bar"}}) tmp_dir.dvc_gen({"foo": "foo", "dir": {"bar": "bar"}})
@@ -927,7 +927,7 @@ def test_filter_targets_not_in_cache(
} }
def test_compat_legacy_new_cache_types(M, tmp_dir, dvc, scm): def test_compat_legacy_new_cache_types(tmp_dir, dvc, scm):
tmp_dir.gen({"foo": "foo", "bar": "bar"}) tmp_dir.gen({"foo": "foo", "bar": "bar"})
(tmp_dir / "foo.dvc").dump( (tmp_dir / "foo.dvc").dump(
{ {
@@ -964,7 +964,7 @@ def test_compat_legacy_new_cache_types(M, tmp_dir, dvc, scm):
} }
def test_missing_cache_remote_check(M, tmp_dir, dvc, scm, local_remote): def test_missing_cache_remote_check(tmp_dir, dvc, scm, local_remote):
tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}}) tmp_dir.dvc_gen({"dir": {"foo": "foo", "bar": "bar"}})
tmp_dir.dvc_gen("foobar", "foobar") tmp_dir.dvc_gen("foobar", "foobar")
remove(dvc.cache.repo.path) remove(dvc.cache.repo.path)
@@ -1046,7 +1046,7 @@ def test_missing_cache_remote_check(M, tmp_dir, dvc, scm, local_remote):
{"committed": {"deleted": [join("dir", "")]}}, {"committed": {"deleted": [join("dir", "")]}},
{ {
"committed": { "committed": {
"deleted": m.unordered( "deleted": M.unordered(
join("dir", ""), join("dir", "bar"), join("dir", "foo") join("dir", ""), join("dir", "bar"), join("dir", "foo")
) )
} }
@@ -1060,7 +1060,7 @@ def test_missing_cache_remote_check(M, tmp_dir, dvc, scm, local_remote):
}, },
{ {
"committed": { "committed": {
"added": m.unordered( "added": M.unordered(
join("dir2", ""), join("dir2", ""),
join("dir2", "bar"), join("dir2", "bar"),
join("dir2", "foo"), join("dir2", "foo"),
@@ -1150,12 +1150,12 @@ def test_missing_cache_remote_check(M, tmp_dir, dvc, scm, local_remote):
param( param(
["dir2", "file2"], ["dir2", "file2"],
{ {
"committed": {"added": m.unordered(join("dir2", ""), "file2")}, "committed": {"added": M.unordered(join("dir2", ""), "file2")},
"uncommitted": {"modified": [join("dir2", "")]}, "uncommitted": {"modified": [join("dir2", "")]},
}, },
{ {
"committed": { "committed": {
"added": m.unordered( "added": M.unordered(
join("dir2", ""), join("dir2", ""),
join("dir2", "bar"), join("dir2", "bar"),
join("dir2", "foo"), join("dir2", "foo"),
@@ -1178,7 +1178,7 @@ def test_missing_cache_remote_check(M, tmp_dir, dvc, scm, local_remote):
}, },
{ {
"committed": { "committed": {
"added": m.unordered( "added": M.unordered(
join("dir2", ""), join("dir2", "bar"), join("dir2", "foo") join("dir2", ""), join("dir2", "bar"), join("dir2", "foo")
), ),
"deleted": ["file"], "deleted": ["file"],
@@ -1193,9 +1193,7 @@ def test_missing_cache_remote_check(M, tmp_dir, dvc, scm, local_remote):
), ),
], ],
) )
def test_renames( def test_renames(tmp_dir, scm, dvc, targets, expected_non_granular, expected_granular):
M, tmp_dir, scm, dvc, targets, expected_non_granular, expected_granular
):
tmp_dir.dvc_gen( tmp_dir.dvc_gen(
{"dir": {"foo": "foo", "bar": "bar"}, "file": "file"}, commit="add dir and file" {"dir": {"foo": "foo", "bar": "bar"}, "file": "file"}, commit="add dir and file"
) )
@@ -1211,3 +1209,62 @@ def test_renames(
dvc.data_status(granular=True, with_renames=True, targets=targets) dvc.data_status(granular=True, with_renames=True, targets=targets)
== EMPTY_STATUS | {"git": M.dict()} | expected_granular == EMPTY_STATUS | {"git": M.dict()} | expected_granular
) )
def test_shallow_should_iterate_upto_tracked_directory(tmp_dir, dvc, scm, local_remote):
"""Testing regression for https://github.com/treeverse/dvc/issues/10899."""
tmp_dir.scm_gen({"dir": {".gitkeep": ""}}, commit="mk dir")
dataset_root = join("dir", "data", "")
tmp_dir.dvc_gen({dataset_root: {"foo": "foo", "bar": "bar"}})
files = [dataset_root, join(dataset_root, "foo"), join(dataset_root, "bar")]
assert dvc.data_status(not_in_remote=True) == {
**EMPTY_STATUS,
"committed": {"added": [dataset_root]},
"not_in_remote": [dataset_root],
"git": M.dict(),
}
assert dvc.data_status(granular=True, not_in_remote=True) == {
**EMPTY_STATUS,
"committed": {"added": M.unordered(*files)},
"not_in_remote": M.unordered(*files),
"git": M.dict(),
}
assert dvc.push() == 3
assert dvc.data_status(not_in_remote=True, remote_refresh=True) == {
**EMPTY_STATUS,
"committed": {"added": [dataset_root]},
"git": M.dict(),
}
assert dvc.data_status(granular=True, not_in_remote=True, remote_refresh=True) == {
**EMPTY_STATUS,
"committed": {"added": M.unordered(*files)},
"git": M.dict(),
}
def test_duplicate_hashes_not_in_remote(tmp_dir, dvc, scm, local_remote):
"""Test that files with identical content (same hash) are all correctly reported.
Regression test for https://github.com/iterative/dvc/issues/10959
"""
tmp_dir.dvc_gen(
{"foo": "content", "bar": "content", "foobar": "foobar"}, commit="add files"
)
assert dvc.data_status(not_in_remote=True, granular=True, remote_refresh=True) == {
**EMPTY_STATUS,
"unchanged": M.unordered("foo", "bar", "foobar"),
"not_in_remote": M.unordered("foo", "bar", "foobar"),
"git": M.dict(),
}
dvc.push()
assert dvc.data_status(not_in_remote=True, granular=True, remote_refresh=True) == {
**EMPTY_STATUS,
"unchanged": M.unordered("foo", "bar", "foobar"),
"not_in_remote": [],
"git": M.dict(),
}
+1 -1
View File
@@ -101,7 +101,7 @@ def test_pull_subdir_file(tmp_dir, erepo_dir):
def test_relative_remote(erepo_dir, tmp_dir): def test_relative_remote(erepo_dir, tmp_dir):
# these steps reproduce the script on this issue: # these steps reproduce the script on this issue:
# https://github.com/iterative/dvc/issues/2756 # https://github.com/treeverse/dvc/issues/2756
with erepo_dir.chdir(): with erepo_dir.chdir():
erepo_dir.dvc_gen("file", "contents", commit="create file") erepo_dir.dvc_gen("file", "contents", commit="create file")
+49 -2
View File
@@ -92,6 +92,27 @@ def test_gc_multiple_dvc_repos(tmp_dir, scm, dvc, erepo_dir):
assert len(list(odb.all())) == 2 assert len(list(odb.all())) == 2
def test_gc_multiple_dvc_repos_all_branches_num(tmp_dir, scm, dvc, erepo_dir):
tmp_dir.dvc_gen("main", "main", commit="main")
erepo_dir.dvc.cache.local.path = dvc.cache.local.path
with erepo_dir.chdir():
erepo_dir.scm_gen("base", "base", commit="base")
with erepo_dir.branch("feature", new=True):
erepo_dir.dvc_gen("shared", "feature-v1", commit="feature-v1")
erepo_dir.dvc.remove("shared.dvc")
erepo_dir.dvc_gen("shared", "feature-v2", commit="feature-v2")
odb = dvc.cache.local
assert len(list(odb.all())) == 3
dvc.gc(all_branches=True, num=2, repos=[erepo_dir])
assert len(list(odb.all())) == 3
dvc.gc(all_branches=True, repos=[erepo_dir])
assert len(list(odb.all())) == 2
def test_all_commits(tmp_dir, scm, dvc): def test_all_commits(tmp_dir, scm, dvc):
tmp_dir.dvc_gen("testfile", "uncommitted") tmp_dir.dvc_gen("testfile", "uncommitted")
tmp_dir.dvc_gen("testfile", "committed", commit="committed") tmp_dir.dvc_gen("testfile", "committed", commit="committed")
@@ -437,8 +458,16 @@ def test_gc_logging(caplog, dvc, good_and_bad_cache):
with caplog.at_level(logging.INFO, logger="dvc"): with caplog.at_level(logging.INFO, logger="dvc"):
dvc.gc(workspace=True) dvc.gc(workspace=True)
assert "Removed 3 objects from repo cache." in caplog.text # Check that 3 objects were removed from either repo or local cache
assert "No unused 'local' cache to remove." in caplog.text # (they point to the same ODB, so only one will be logged due to deduplication)
has_repo_log = "Removed 3 objects from repo cache." in caplog.text
has_local_log = "Removed 3 objects from local cache." in caplog.text
# Exactly one of repo or local should have the log, not both (deduplication)
assert has_repo_log ^ has_local_log, (
"Should have exactly one log for repo/local cache"
)
assert "No unused 'legacy' cache to remove." in caplog.text assert "No unused 'legacy' cache to remove." in caplog.text
@@ -450,3 +479,21 @@ def test_gc_skip_failed(tmp_dir, dvc):
dvc.gc(force=True, workspace=True) dvc.gc(force=True, workspace=True)
dvc.gc(force=True, workspace=True, skip_failed=True) dvc.gc(force=True, workspace=True, skip_failed=True)
def test_gc_dry_logs_paths(caplog, tmp_dir, dvc):
"""Test that dry run logs paths of objects to be removed."""
odb = dvc.cache.local
# Add some unused objects directly to cache
unused_hashes = ["test123", "test456", "test789"]
for hash_val in unused_hashes:
odb.add_bytes(hash_val, hash_val.encode("utf8"))
with caplog.at_level(logging.INFO, logger="dvc_data.hashfile.gc"):
dvc.gc(workspace=True, dry=True)
# Verify that paths are logged by dvc-data layer
for hash_val in unused_hashes:
expected_path = odb.oid_to_path(hash_val)
assert f"Removing {expected_path}" in caplog.text
+1 -1
View File
@@ -160,7 +160,7 @@ def test_non_cached_output(tmp_dir, erepo_dir):
assert (tmp_dir / dst).read_text().strip() == "hello" assert (tmp_dir / dst).read_text().strip() == "hello"
# https://github.com/iterative/dvc/pull/2837#discussion_r352123053 # https://github.com/treeverse/dvc/pull/2837#discussion_r352123053
def test_absolute_file_outside_repo(tmp_dir, erepo_dir): def test_absolute_file_outside_repo(tmp_dir, erepo_dir):
with pytest.raises(FileNotFoundError): with pytest.raises(FileNotFoundError):
Repo.get(os.fspath(erepo_dir), "/root/") Repo.get(os.fspath(erepo_dir), "/root/")
+1 -1
View File
@@ -405,7 +405,7 @@ def test_import_non_existing(erepo_dir, tmp_dir, dvc):
with pytest.raises(FileNotFoundError): with pytest.raises(FileNotFoundError):
tmp_dir.dvc.imp(os.fspath(erepo_dir), "invalid_output") tmp_dir.dvc.imp(os.fspath(erepo_dir), "invalid_output")
# https://github.com/iterative/dvc/pull/2837#discussion_r352123053 # https://github.com/treeverse/dvc/pull/2837#discussion_r352123053
with pytest.raises(FileNotFoundError): with pytest.raises(FileNotFoundError):
tmp_dir.dvc.imp(os.fspath(erepo_dir), "/root/", "root") tmp_dir.dvc.imp(os.fspath(erepo_dir), "/root/", "root")
+3 -1
View File
@@ -6,6 +6,8 @@ import pandas as pd
import pytest import pytest
from funcy import compact from funcy import compact
from dvc.testing import matchers as M
@pytest.fixture @pytest.fixture
def db_path(tmp_dir): def db_path(tmp_dir):
@@ -46,7 +48,7 @@ def load_data(file, output_format):
({"table": "model"}, "model"), ({"table": "model"}, "model"),
], ],
) )
def test(M, tmp_dir, scm, dvc, db_connection, seed_db, output_format, args, file_name): def test(tmp_dir, scm, dvc, db_connection, seed_db, output_format, args, file_name):
seed_db(values=range(5)) seed_db(values=range(5))
if output_format == "json": if output_format == "json":
file_size = 96, 192 file_size = 96, 192
+1 -1
View File
@@ -109,7 +109,7 @@ def test_gen_dvcignore(tmp_dir):
def test_init_when_ignored_by_git(tmp_dir, scm, caplog): def test_init_when_ignored_by_git(tmp_dir, scm, caplog):
# https://github.com/iterative/dvc/issues/3738 # https://github.com/treeverse/dvc/issues/3738
tmp_dir.gen({".gitignore": ".*"}) tmp_dir.gen({".gitignore": ".*"})
with caplog.at_level(logging.ERROR, logger="dvc"): with caplog.at_level(logging.ERROR, logger="dvc"):
assert main(["init"]) == 1 assert main(["init"]) == 1
+7 -6
View File
@@ -10,6 +10,7 @@ from dvc.fs import MemoryFileSystem
from dvc.repo import Repo from dvc.repo import Repo
from dvc.repo.ls import _ls_tree, ls_tree from dvc.repo.ls import _ls_tree, ls_tree
from dvc.scm import CloneError from dvc.scm import CloneError
from dvc.testing import matchers as M
FS_STRUCTURE = { FS_STRUCTURE = {
"README.md": "content", "README.md": "content",
@@ -460,7 +461,7 @@ def test_ls_shows_pipeline_tracked_outs(tmp_dir, dvc, scm, run_copy):
match_files(files, ((("bar",), True),)) match_files(files, ((("bar",), True),))
def test_ls_granular(erepo_dir, M): def test_ls_granular(erepo_dir):
with erepo_dir.chdir(): with erepo_dir.chdir():
erepo_dir.dvc_gen( erepo_dir.dvc_gen(
{ {
@@ -624,7 +625,7 @@ def test_subrepo(request, dvc_top_level, erepo_type):
assert _list_files(subrepo, "dvc_dir") == {"lorem"} assert _list_files(subrepo, "dvc_dir") == {"lorem"}
def test_broken_symlink(tmp_dir, dvc, M): def test_broken_symlink(tmp_dir, dvc):
from dvc.fs import system from dvc.fs import system
tmp_dir.gen("file", "content") tmp_dir.gen("file", "content")
@@ -654,7 +655,7 @@ def test_broken_symlink(tmp_dir, dvc, M):
] ]
def test_ls_broken_dir(tmp_dir, dvc, M): def test_ls_broken_dir(tmp_dir, dvc):
from dvc_data.index import DataIndexDirError from dvc_data.index import DataIndexDirError
tmp_dir.dvc_gen( tmp_dir.dvc_gen(
@@ -836,7 +837,7 @@ def _simplify_tree(files):
return ret return ret
def test_ls_tree(M, tmp_dir, scm, dvc): def test_ls_tree(tmp_dir, scm, dvc):
tmp_dir.scm_gen(FS_STRUCTURE, commit="init") tmp_dir.scm_gen(FS_STRUCTURE, commit="init")
tmp_dir.dvc_gen(DVC_STRUCTURE, commit="dvc") tmp_dir.dvc_gen(DVC_STRUCTURE, commit="dvc")
@@ -887,7 +888,7 @@ def test_ls_tree(M, tmp_dir, scm, dvc):
} }
def test_ls_tree_dvc_only(M, tmp_dir, scm, dvc): def test_ls_tree_dvc_only(tmp_dir, scm, dvc):
tmp_dir.scm_gen(FS_STRUCTURE, commit="init") tmp_dir.scm_gen(FS_STRUCTURE, commit="init")
tmp_dir.dvc_gen(DVC_STRUCTURE, commit="dvc") tmp_dir.dvc_gen(DVC_STRUCTURE, commit="dvc")
@@ -905,7 +906,7 @@ def test_ls_tree_dvc_only(M, tmp_dir, scm, dvc):
assert _simplify_tree(files) == expected assert _simplify_tree(files) == expected
def test_ls_tree_maxdepth(M, tmp_dir, scm, dvc): def test_ls_tree_maxdepth(tmp_dir, scm, dvc):
tmp_dir.scm_gen(FS_STRUCTURE, commit="init") tmp_dir.scm_gen(FS_STRUCTURE, commit="init")
tmp_dir.dvc_gen(DVC_STRUCTURE, commit="dvc") tmp_dir.dvc_gen(DVC_STRUCTURE, commit="dvc")
+201 -4
View File
@@ -6,14 +6,20 @@ import pytest
from dvc.cli import main from dvc.cli import main
from dvc.exceptions import MoveNotDataSourceError, OutputNotFoundError from dvc.exceptions import MoveNotDataSourceError, OutputNotFoundError
from dvc.stage.exceptions import StageFileAlreadyExistsError
def test_move(tmp_dir, dvc): def test_move(tmp_dir, dvc, scm):
tmp_dir.dvc_gen("foo", "foo") tmp_dir.dvc_gen("foo", "bar")
dvc.move("foo", "foo1") assert (tmp_dir / "foo.dvc").exists()
dvc.move("foo", "bar")
assert not (tmp_dir / "foo.dvc").exists()
assert (tmp_dir / "bar.dvc").exists()
assert not (tmp_dir / "foo").is_file() assert not (tmp_dir / "foo").is_file()
assert (tmp_dir / "foo1").is_file() assert (tmp_dir / "bar").is_file()
# should only have the new path in the .gitignore, and only once
assert (tmp_dir / ".gitignore").read_text().splitlines() == ["/bar"]
def test_move_non_existent_file(dvc): def test_move_non_existent_file(dvc):
@@ -21,6 +27,24 @@ def test_move_non_existent_file(dvc):
dvc.move("non_existent_file", "dst") dvc.move("non_existent_file", "dst")
def test_move_missing_file(tmp_dir, dvc, scm, caplog):
tmp_dir.dvc_gen("foo", "foo")
(tmp_dir / "foo").unlink()
contents = (tmp_dir / "foo.dvc").parse()
dvc.move("foo", "bar")
assert not (tmp_dir / "foo.dvc").exists()
# only the path should be changed in the dvc file
contents["outs"][0]["path"] = "bar"
assert contents == (tmp_dir / "bar.dvc").parse()
# file should not be checked out
assert not (tmp_dir / "foo").is_file()
assert not (tmp_dir / "bar").is_file()
# should only have the new path in the .gitignore, and only once
assert (tmp_dir / ".gitignore").read_text().splitlines() == ["/bar"]
def test_move_directory(tmp_dir, dvc): def test_move_directory(tmp_dir, dvc):
tmp_dir.dvc_gen("data", {"foo": "foo", "bar": "bar"}) tmp_dir.dvc_gen("data", {"foo": "foo", "bar": "bar"})
dvc.move("data", "dst") dvc.move("data", "dst")
@@ -195,3 +219,176 @@ def test_move_meta(tmp_dir, dvc):
custom_key: 42 custom_key: 42
""" """
) )
def test_import(tmp_dir, dvc, scm):
tmp_dir.dvc_gen("foo", "foo", commit="add foo")
imp_stage = dvc.imp(os.curdir, "foo", "foo_imported")
dvc.move("foo_imported", "foo_moved")
(stage,) = dvc.stage.collect("foo_moved.dvc")
assert imp_stage.md5 != stage.md5
res = (tmp_dir / "foo_moved.dvc").read_text()
assert res == textwrap.dedent(
f"""\
md5: {stage.md5}
frozen: true
deps:
- path: foo
repo:
url: {os.curdir}
rev_lock: {scm.get_rev()}
outs:
- md5: acbd18db4cc2f85cedef654fccc4a4d8
size: 3
hash: md5
path: foo_moved
"""
)
@pytest.mark.parametrize(
"path_func",
[pytest.param(os.path.abspath, id="abs"), pytest.param(os.path.relpath, id="rel")],
)
def test_import_url_in_repo(tmp_dir, dvc, path_func):
tmp_dir.gen("foo", "foo")
imp_stage = dvc.imp_url(path_func(tmp_dir / "foo"), "foo_imported")
(tmp_dir / "data").mkdir()
dvc.move("foo_imported", os.path.join("data", "foo_moved"))
(stage,) = dvc.stage.collect(os.path.join("data", "foo_moved.dvc"))
assert imp_stage.md5 != stage.md5
res = (tmp_dir / "data" / "foo_moved.dvc").read_text()
assert res == textwrap.dedent(
f"""\
md5: {stage.md5}
frozen: true
deps:
- md5: acbd18db4cc2f85cedef654fccc4a4d8
size: 3
hash: md5
path: ../foo
outs:
- md5: acbd18db4cc2f85cedef654fccc4a4d8
size: 3
hash: md5
path: foo_moved
"""
)
@pytest.mark.parametrize(
"path_func",
[pytest.param(os.path.abspath, id="abs"), pytest.param(os.path.relpath, id="rel")],
)
def test_import_url_out_of_repo(tmp_dir, dvc, scm, path_func, make_tmp_dir):
external = make_tmp_dir("external")
external.gen("foo", "foo")
imp_stage = dvc.imp_url(path_func(external / "foo"), "foo_imported")
data_dir = tmp_dir / "data"
data_dir.mkdir()
new_path = data_dir / "foo_moved"
new_dvcfile = new_path.with_suffix(".dvc")
dvc.move("foo_imported", os.fspath(new_path))
(stage,) = dvc.stage.collect(os.fspath(new_dvcfile))
assert imp_stage.md5 != stage.md5
with data_dir.chdir():
expected_path = path_func(external / "foo")
assert new_dvcfile.parse() == {
"md5": stage.md5,
"frozen": True,
"deps": [
{
"md5": "acbd18db4cc2f85cedef654fccc4a4d8",
"size": 3,
"hash": "md5",
"path": expected_path,
}
],
"outs": [
{
"md5": "acbd18db4cc2f85cedef654fccc4a4d8",
"size": 3,
"hash": "md5",
"path": "foo_moved",
}
],
}
@pytest.mark.parametrize(
"path_func",
[pytest.param(os.path.abspath, id="abs"), pytest.param(os.path.relpath, id="rel")],
)
def test_all_metadata_are_preserved(tmp_dir, dvc, make_tmp_dir, path_func):
external = make_tmp_dir("external")
external.gen("foo", "foo")
contents = {
"md5": "bad", # placeholder, does not matter for the test
"frozen": True,
"desc": "this is a stage description",
"always_changed": True,
"meta": {"custom_key": 42},
"deps": [
{
"md5": "acbd18db4cc2f85cedef654fccc4a4d8",
"size": 3,
"hash": "md5",
"path": path_func(external / "foo"),
}
],
"outs": [
{
"md5": "acbd18db4cc2f85cedef654fccc4a4d8",
"path": "foo_imported",
"persist": True,
"hash": "md5",
"size": 3,
"desc": "this is a description",
"type": "model",
"labels": ["label1", "label2"],
"meta": {"custom_key": 42},
"cache": False,
"remote": "myremote",
"push": False,
}
],
}
(tmp_dir / "foo_imported.dvc").dump(contents)
(tmp_dir / "foo_imported").write_text("foo")
data_dir = tmp_dir / "data"
data_dir.mkdir()
new_path = data_dir / "foo_moved"
new_dvcfile = new_path.with_suffix(".dvc")
dvc.move("foo_imported", os.fspath(new_path))
(stage,) = dvc.stage.collect(os.fspath(new_dvcfile))
with data_dir.chdir():
expected_path = path_func(external / "foo")
contents["outs"][0] |= {"path": "foo_moved"}
contents["deps"][0] |= {"path": expected_path}
contents |= {"md5": stage.md5}
assert new_dvcfile.parse() == contents
def test_move_dst_stage_file_already_exists(tmp_dir, dvc):
tmp_dir.dvc_gen({"foo": "foo", "bar": "bar"})
with pytest.raises(StageFileAlreadyExistsError) as exc_info:
dvc.move("foo", "bar")
assert str(exc_info.value) == "'bar.dvc' already exists"
assert exc_info.value.__cause__ is None
+1 -1
View File
@@ -125,7 +125,7 @@ def assert_index_equal(first, second, strict=True, ordered=True):
def test_skip_graph_checks(dvc, mocker): def test_skip_graph_checks(dvc, mocker):
# See https://github.com/iterative/dvc/issues/2671 for more info # See https://github.com/treeverse/dvc/issues/2671 for more info
mock_build_graph = mocker.spy(Index.graph, "fget") mock_build_graph = mocker.spy(Index.graph, "fget")
# sanity check # sanity check
+1 -1
View File
@@ -166,7 +166,7 @@ def test_virtual_update_noop(tmp_dir, dvc, remote):
assert not dvc.push() assert not dvc.push()
def test_partial_checkout_and_update(M, tmp_dir, dvc, remote): def test_partial_checkout_and_update(tmp_dir, dvc, remote):
dir1 = {f"{i}.txt": f"dir1 {i}" for i in range(10)} dir1 = {f"{i}.txt": f"dir1 {i}" for i in range(10)}
dir2 = {f"{i}.txt": f"dir2 {i}" for i in range(10)} dir2 = {f"{i}.txt": f"dir2 {i}" for i in range(10)}
tmp_dir.gen({"dir": {"foo": "foo", "subdir": dir1, "subdir2": dir2}}) tmp_dir.gen({"dir": {"foo": "foo", "subdir": dir1, "subdir2": dir2}})
+2 -2
View File
@@ -241,7 +241,7 @@ def test_compose_and_dump_dir_module(
def test_compose_and_dump_yaml_handles_string(tmp_dir): def test_compose_and_dump_yaml_handles_string(tmp_dir):
"""Regression test for https://github.com/iterative/dvc/issues/8583""" """Regression test for https://github.com/treeverse/dvc/issues/8583"""
from dvc.utils.hydra import compose_and_dump from dvc.utils.hydra import compose_and_dump
config = tmp_dir / "conf" / "config.yaml" config = tmp_dir / "conf" / "config.yaml"
@@ -253,7 +253,7 @@ def test_compose_and_dump_yaml_handles_string(tmp_dir):
def test_compose_and_dump_resolves_interpolation(tmp_dir): def test_compose_and_dump_resolves_interpolation(tmp_dir):
"""Regression test for https://github.com/iterative/dvc/issues/9196""" """Regression test for https://github.com/treeverse/dvc/issues/9196"""
from dvc.utils.hydra import compose_and_dump from dvc.utils.hydra import compose_and_dump
config = tmp_dir / "conf" / "config.yaml" config = tmp_dir / "conf" / "config.yaml"
+1 -1
View File
@@ -59,7 +59,7 @@ def test_remote_missing_deps_are_correctly_reported(tmp_dir, caplog, mocker, pkg
else: else:
expected += ( expected += (
"\nPlease report this bug to " "\nPlease report this bug to "
"<https://github.com/iterative/dvc/issues>. " "<https://github.com/treeverse/dvc/issues>. "
"Thank you!" "Thank you!"
) )
assert expected in caplog.text assert expected in caplog.text
+53 -2
View File
@@ -2,7 +2,15 @@ import networkx as nx
import pytest import pytest
from dvc.cli import main, parse_args from dvc.cli import main, parse_args
from dvc.commands.dag import CmdDAG, _build, _show_ascii, _show_dot, _show_mermaid from dvc.commands.dag import (
CmdDAG,
_build,
_collapse_foreach_matrix,
_show_ascii,
_show_dot,
_show_mermaid,
)
from dvc.parsing import JOIN
@pytest.mark.parametrize( @pytest.mark.parametrize(
@@ -49,10 +57,53 @@ def repo(tmp_dir, dvc):
return dvc return dvc
def test_collapse_foreach_matrix(repo):
graph = nx.DiGraph(
[
("2", "1"),
("3@a", "2"),
("3@a", "1"),
("4", "1"),
("3@b", "4"),
("3@b", "1"),
("5", "3@a"),
("6", "3@b"),
("7", "5"),
("7", "6"),
]
)
expected_graph = nx.DiGraph(
[
("2", "1"),
("3", "2"),
("4", "1"),
("3", "4"),
("3", "1"),
("5", "3"),
("6", "3"),
("7", "5"),
("7", "6"),
]
)
collapsed_graph = _collapse_foreach_matrix(graph)
for node in collapsed_graph.nodes:
assert JOIN not in node
for n1, n2 in collapsed_graph.edges:
assert JOIN not in n1
assert JOIN not in n2
assert nx.is_isomorphic(collapsed_graph, expected_graph)
def test_build(repo): def test_build(repo):
assert nx.is_isomorphic(_build(repo), repo.index.graph) assert nx.is_isomorphic(_build(repo), repo.index.graph)
def test_build_collapse(repo):
assert nx.is_isomorphic(
_build(repo, collapse_foreach_matrix=True), repo.index.graph
)
def test_build_target(repo): def test_build_target(repo):
graph = _build(repo, target="3") graph = _build(repo, target="3")
assert set(graph.nodes()) == {"3", "b.dvc", "a.dvc"} assert set(graph.nodes()) == {"3", "b.dvc", "a.dvc"}
@@ -119,7 +170,7 @@ def test_show_ascii(repo):
def test_show_dot(repo): def test_show_dot(repo):
# dot file rendering is not deterministic though graph # dot file rendering is not deterministic though graph
# output doesn't depend upon order of lines. Use sorted values # output doesn't depend upon order of lines. Use sorted values
# https://github.com/iterative/dvc/pull/7725 # https://github.com/treeverse/dvc/pull/7725
expected = [ expected = [
"\"stage: '1'\";", "\"stage: '1'\";",
"\"stage: '2'\";", "\"stage: '2'\";",
+1 -1
View File
@@ -196,7 +196,7 @@ def test_diff_show_markdown_and_hash(mocker, show_hash, dvc):
cmd = args.func(args) cmd = args.func(args)
diff = {} diff = {}
show_hash = show_hash if show_hash else False show_hash = show_hash or False
mock_show_markdown = mocker.patch("dvc.commands.diff._show_markdown") mock_show_markdown = mocker.patch("dvc.commands.diff._show_markdown")
mocker.patch("dvc.repo.Repo.diff", return_value=diff.copy()) mocker.patch("dvc.repo.Repo.diff", return_value=diff.copy())
+2
View File
@@ -130,6 +130,7 @@ def test_experiments_run(dvc, scm, mocker):
"tmp_dir": False, "tmp_dir": False,
"copy_paths": [], "copy_paths": [],
"message": None, "message": None,
"no_hydra": False,
} }
default_arguments.update(repro_arguments) default_arguments.update(repro_arguments)
@@ -151,6 +152,7 @@ def test_experiments_run_message(dvc, scm, mocker, flag):
"tmp_dir": False, "tmp_dir": False,
"copy_paths": [], "copy_paths": [],
"message": "mymessage", "message": "mymessage",
"no_hydra": False,
} }
default_arguments.update(repro_arguments) default_arguments.update(repro_arguments)

Some files were not shown because too many files have changed in this diff Show More