[pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.15.20 → v0.16.10](https://github.com/astral-sh/ruff-pre-commit/compare/v0.15.20...v0.16.10)
- [github.com/codespell-project/codespell: v2.4.2 → v2.4.3](https://github.com/codespell-project/codespell/compare/v2.4.2...v2.4.3)

.github/ISSUE_TEMPLATE/bug_report.md

@@ -57,7 +57,7 @@ $ dvc doctor
 **Additional Information (if any):**
 
 <!--
-Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.
+Please check https://github.com/treeverse/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.
 
 If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
 If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.

.github/ISSUE_TEMPLATE/epic_story.md

@@ -1,60 +0,0 @@
----
-name: Epic/Story Issue Template
-about: Template for "top" level issues - Epics (>2 weeks) / Stories (<2 weeks)
-title: 'Epic: New Feature'
-labels: epic
-assignees:
-
----
-
-## Summary / Background
-What do you want to achieve, why? business context
-...
-
-## Scope
-
-What will be impacted and what won't be?
-What needs implementation and what is invariant?
-e.g.
-- user should be able to run workflow X from UI
-- enable workflow Y, Z from CLI
-
-## Assumptions
-Product / UX assumptions as well as technical assumptions / limitations
-e.g.
-* Support only Python Runtime
-* Focus on DVC experiments only
-* Deployment environments don't change often and can be picked up from shared configuration
-
-## Open Questions
-e.g.
-- How should access control work for shared artifacts (workflow X)
-- Python runtime assumption - is it really valid? in light of <...>
-
-## Blockers / Dependencies
-List issues or other conditions / blockers
-
-## General Approach
-Invocation example:
-```shell
-$ mapper-run task.tar.gz --ray-cluster <ip>:<port>
-```
-
-## Steps
-
-### Must have (p1)
-- [ ] subissue2
-- [ ] step 2
- - info
- - info
-
-### Optional / followup (p2)
-- [ ] ⌛ step 3 wip
-- [ ] step 4
-
-## Timelines
-
-Put your estimations here. Update once certainty changes
-- end of week (Feb 3) for prototype with workflows X, Y
-- Feb 15 - MVP in production
-- Low priority followups can be done later

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,5 +1,5 @@
 * [ ] ❗ I have followed the [Contributing to DVC](https://dvc.org/doc/user-guide/contributing/core) checklist.
 
-* [ ] 📖 If this PR requires [documentation](https://dvc.org/doc) updates, I have created a separate PR (or issue, at least) in [dvc.org](https://github.com/iterative/dvc.org) and linked it here.
+* [ ] 📖 If this PR requires [documentation](https://dvc.org/doc) updates, I have created a separate PR (or issue, at least) in [dvc.org](https://github.com/treeverse/dvc.org) and linked it here.
 
 Thank you for the contribution - we'll try to review it as soon as possible. 🙏

.github/mergify.yml

@@ -1,10 +0,0 @@
-pull_request_rules:
-  - name: backport patches to 2.x
-    conditions:
-      - base=main
-      - label=backport
-    actions:
-      backport:
-        branches:
-          - "2.x"
-        title: ({{ destination_branch }}) {{ title }}

.github/workflows/benchmarks.yaml

@@ -5,14 +5,40 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
 
+env:
+  FORCE_COLOR: "1"
+  PY_COLORS: "1"
+
 jobs:
   bench:
     name: run benchmarks
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/setup-python@v5
+      - uses: actions/setup-python@v7
         with:
-            python-version: "3.11"
-      - uses: iterative/dvc-bench@main
+            python-version: "3.14"
+
+      - uses: actions/checkout@v7
         with:
-            pytest_options: "-k 'test_init or test_help'"
+          ref: ${{ github.event.pull_request.base.sha }}
+          fetch-depth: 0
+
+      - uses: astral-sh/setup-uv@v7
+        with:
+          enable-cache: true
+          cache-dependency-glob: pyproject.toml
+
+      - run: uv pip install '.[tests]' --system
+      - name: run benchmarks on base branch
+        run: pytest --benchmark-autosave dvc/testing/benchmarks/ -k 'test_init or test_help'
+
+      - uses: actions/checkout@v7
+        with:
+          fetch-depth: 0
+          clean: false
+      - run: uv pip install '.[tests]' --system
+      - name: run benchmarks for PR
+        run: >
+          pytest --benchmark-compare --benchmark-compare-fail=min:5%
+          --benchmark-group-by name
+          dvc/testing/benchmarks/ -k 'test_init or test_help'

.github/workflows/build.yaml

@@ -19,46 +19,49 @@ jobs:
     name: Build & Verify package
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v7
       with:
         fetch-depth: 0
 
-    - uses: actions/setup-python@v5
+    - uses: actions/setup-python@v7
       with:
-        python-version: 3.8
+        python-version: "3.14"
 
-    - run: python -m pip install --upgrade pip setuptools_scm build twine
+    - uses: astral-sh/setup-uv@v7
 
-    - name: Force version for Test PyPI upload
-      if: ${{ !startsWith(github.ref, 'refs/tags') }}
-      run: echo version=$(python -m setuptools_scm | awk -F+ '{print $1}' | tail -1) >> $GITHUB_ENV
+    - name: Omit local version for Test PyPI upload
+      if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+      run: echo SETUPTOOLS_SCM_OVERRIDES_FOR_DVC='{local_scheme="no-local-version"}' >> $GITHUB_ENV
 
-    - run: echo 'PKG = "pip"'>dvc/_build.py
-    - run: python -m build
-      env:
-        SETUPTOOLS_SCM_PRETEND_VERSION_FOR_DVC: ${{ env.version }}
-    - run: twine check --strict dist/*
+    - name: Build Python Package
+      run: |
+        echo 'PKG = "pip"'>dvc/_build.py
+        uv build
 
-    - uses: actions/upload-artifact@v4
+    - name: Check dist
+      run: uv tool run twine check --strict dist/*
+
+    - uses: actions/upload-artifact@v7
       with:
         name: Packages
         path: dist/
+        if-no-files-found: error
 
   test-pypi-publish:
     name: Publish dev package to test.pypi.org
     runs-on: ubuntu-latest
     needs: build
-    if: ${{ github.event.action == 'published' || (github.repository == 'iterative/dvc' && github.event_name == 'push' && github.ref == 'refs/heads/main') }}
+    if: ${{ github.event.action == 'published' || (github.repository == 'treeverse/dvc' && github.event_name == 'push' && github.ref == 'refs/heads/main') }}
 
     environment:
       name: test-pypi
-      url: https://test.pypi.org/p/dvc
+      url: https://test.pypi.org/p/dvc/${{ github.event.release.tag_name }}
 
     permissions:
       id-token: write
 
     steps:
-    - uses: actions/download-artifact@v4
+    - uses: actions/download-artifact@v8
       with:
         name: Packages
         path: dist
@@ -77,13 +80,13 @@ jobs:
 
     environment:
       name: pypi
-      url: https://pypi.org/p/dvc
+      url: https://pypi.org/p/dvc/${{ github.event.release.tag_name }}
 
     permissions:
       id-token: write
 
     steps:
-    - uses: actions/download-artifact@v4
+    - uses: actions/download-artifact@v8
       with:
         name: Packages
         path: dist

.github/workflows/codeql.yml

@@ -2,7 +2,7 @@ name: "CodeQL"
 
 on:
   push:
-    branches: ["main", "2.x"]
+    branches: ["main"]
   pull_request:
     branches: ["main"]
   schedule:
@@ -28,13 +28,13 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v4
+      uses: actions/checkout@v7
 
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v3
+      uses: github/codeql-action/init@v4.37.4
       with:
         languages: ${{ matrix.language }}
         queries: security-extended
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v3
+      uses: github/codeql-action/analyze@v4.37.4

.github/workflows/plugin_tests.yaml

@@ -18,35 +18,36 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-20.04]
-        pyv: ["3.10"]
+        os: [ubuntu-latest]
+        pyv: ["3.14"]
         plugin: ["dvc-s3"]
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v7
       with:
         path: dvc
 
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v7
       with:
-        repository: iterative/${{ matrix.plugin }}
+        repository: treeverse/${{ matrix.plugin }}
         ref: main
         path: ${{ matrix.plugin }}
 
     - name: Set up Python
-      uses: actions/setup-python@v5
+      uses: actions/setup-python@v7
       with:
         python-version: ${{ matrix.pyv }}
-        cache: pip
-        cache-dependency-path: |
-          dvc/pyproject.toml
-          ${{ matrix.plugin }}/setup.cfg
+
+    - uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        cache-dependency-glob: "**/pyproject.toml"
+        cache-suffix: ${{ matrix.pyv }}
 
     - name: Install plugin + DVC@PR
       run: |
-        pip install --upgrade pip wheel
-        pip install "./dvc[testing]"
-        pip install -e "./${{ matrix.plugin }}[tests]"
+        uv pip install "./dvc[testing]" --system
+        uv pip install -e "./${{ matrix.plugin }}[tests]" --system
 
     - name: Run plugin tests
       timeout-minutes: 15

.github/workflows/tests.yaml

@@ -2,7 +2,7 @@ name: Tests
 
 on:
   push:
-    branches: [main, 2.x]
+    branches: [main]
   pull_request:
   schedule:
     - cron: '5 1 * * *'  # every day at 01:05
@@ -16,42 +16,146 @@ permissions:
   contents: read
 
 jobs:
+  lint:
+    timeout-minutes: 10
+    runs-on: ${{ matrix.os }}
+    env:
+       DVC_NO_ANALYTICS: true
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        fetch-depth: 0
+    - name: Set up Python
+      uses: actions/setup-python@v7
+      with:
+        python-version: '3.14'
+    - uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        cache-dependency-glob: pyproject.toml
+    - run: uv pip install -e ".[dev]" --system
+      env:
+        UV_LINK_MODE: ${{ runner.os == 'Windows' && 'symlink' || 'hardlink' }}
+
+    - name: Cache mypy
+      uses: actions/cache@v6
+      with:
+        path: .mypy_cache
+        key: mypy-${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
+    - uses: pre-commit/[email protected]
+
   tests:
-    timeout-minutes: 50
+    timeout-minutes: 20
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-20.04, windows-latest, macos-latest]
-        pyv: ["3.8", "3.9", "3.10", "3.11", "3.12"]
-
+        os: [ubuntu-latest, macos-latest]
+        pyv: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
+        include:
+        - os: windows-latest
+          pyv: "3.9"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 1"
+        - os: windows-latest
+          pyv: "3.9"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 2"
+        - os: windows-latest
+          pyv: "3.9"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 3"
+        - os: windows-latest
+          pyv: "3.9"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 4"
+        - os: windows-latest
+          pyv: "3.10"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 1"
+        - os: windows-latest
+          pyv: "3.10"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 2"
+        - os: windows-latest
+          pyv: "3.10"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 3"
+        - os: windows-latest
+          pyv: "3.10"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 4"
+        - os: windows-latest
+          pyv: "3.11"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 1"
+        - os: windows-latest
+          pyv: "3.11"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 2"
+        - os: windows-latest
+          pyv: "3.11"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 3"
+        - os: windows-latest
+          pyv: "3.11"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 4"
+        - os: windows-latest
+          pyv: "3.12"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 1"
+        - os: windows-latest
+          pyv: "3.12"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 2"
+        - os: windows-latest
+          pyv: "3.12"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 3"
+        - os: windows-latest
+          pyv: "3.12"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 4"
+        - os: windows-latest
+          pyv: "3.13"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 1"
+        - os: windows-latest
+          pyv: "3.13"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 2"
+        - os: windows-latest
+          pyv: "3.13"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 3"
+        - os: windows-latest
+          pyv: "3.13"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 4"
+        - os: windows-latest
+          pyv: "3.14"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 1"
+        - os: windows-latest
+          pyv: "3.14"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 2"
+        - os: windows-latest
+          pyv: "3.14"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 3"
+        - os: windows-latest
+          pyv: "3.14"
+          pytestargs: "--splitting-algorithm=least_duration --splits 4 --group 4"
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v7
       with:
         fetch-depth: 0
     - name: Set up Python
-      uses: actions/setup-python@v5
+      uses: actions/setup-python@v7
       with:
         python-version: ${{ matrix.pyv }}
-        cache: pip
-        cache-dependency-path: pyproject.toml
-    - name: install
-      run: |
-        pip install --upgrade pip wheel
-        pip install -e ".[dev]"
-    - uses: pre-commit/[email protected]
+    - uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        cache-dependency-glob: pyproject.toml
+        cache-suffix: ${{ matrix.pyv }}
+    - run: uv pip install -e ".[dev]" pytest-split --system
+      env:
+        UV_LINK_MODE: ${{ runner.os == 'Windows' && 'symlink' || 'hardlink' }}
+
     - name: run tests
       timeout-minutes: 40
       env:
+        COVERAGE_CORE: sysmon
         PYTHONUTF8: 1
       run: >
-        pytest -n=logical --timeout=300 --durations=100
-        --cov --cov-report=xml --cov-report=term
+        pytest ${{ matrix.pytestargs }} -n=logical --dist=worksteal --timeout=300 --durations=0
+        --cov --cov-report=xml --cov-report=term  --durations-path=./.github/.test_durations
     - name: upload coverage report
-      uses: codecov/codecov-action@v3
-      with:
-        file: ./coverage.xml
-        fail_ci_if_error: false
+      uses: codecov/codecov-action@v7
   check:
     if: always()
     needs: [tests]
@@ -66,7 +170,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Slack Notification
-      uses: rtCamp/action-slack-notify@v2.2.1
+      uses: rtCamp/action-slack-notify@v2.4.0
       env:
         SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
         SLACK_COLOR: ${{ job.status }}

.gitignore

@@ -43,7 +43,7 @@ scripts/ci/gcp-creds.json
 .gcp-creds.json
 
 *~
-/dvc/_dvc_version.py
+/dvc/_version.py
 
 .mypy_cache/
 .pytest_cache/

.pre-commit-config.yaml

@@ -6,7 +6,7 @@ ci:
 
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.5.0
+    rev: v6.0.0
     hooks:
       - id: check-added-large-files
       - id: check-case-conflict
@@ -22,13 +22,13 @@ repos:
       - id: sort-simple-yaml
       - id: trailing-whitespace
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: 'v0.1.11'
+    rev: 'v0.16.10'
     hooks:
-      - id: ruff
+      - id: ruff-check
         args: [--fix, --exit-non-zero-on-fix]
       - id: ruff-format
   - repo: https://github.com/codespell-project/codespell
-    rev: v2.2.6
+    rev: v2.4.3
     hooks:
       - id: codespell
         additional_dependencies: ["tomli"]
@@ -50,7 +50,7 @@ repos:
         language: system
         name: DVC pre-commit
         stages:
-          - commit
+          - pre-commit
         verbose: true
         require_serial: true
       - args:
@@ -61,7 +61,7 @@ repos:
         language: system
         name: DVC pre-push
         stages:
-          - push
+          - pre-push
         require_serial: true
       - always_run: true
         args:

.pre-commit-hooks.yaml

@@ -8,7 +8,7 @@
   name: DVC pre-commit
   require_serial: true
   stages:
-  - commit
+  - pre-commit
   verbose: true
 - args:
   - git-hook
@@ -20,7 +20,7 @@
   name: DVC pre-push
   require_serial: true
   stages:
-  - push
+  - pre-push
 - always_run: true
   args:
   - git-hook
@@ -29,7 +29,7 @@
   id: dvc-post-checkout
   language: python
   language_version: python3
-  minimum_pre_commit_version: 2.2.0
+  minimum_pre_commit_version: 3.2.0
   name: DVC post-checkout
   require_serial: true
   stages:

.zenodo.json

@@ -4,5 +4,5 @@
     "data-science", "data-version-control", "machine-learning", "git",
     "developer-tools", "reproducibility", "collaboration", "ai", "python"],
   "contributors": [
-    {"name": "DVC team", "type": "Other", "affiliation": "Iterative"}]
+    {"name": "DVC team", "type": "Other", "affiliation": "Treeverse"}]
 }

CITATION.cff

@@ -0,0 +1,25 @@
+cff-version: 1.2.0
+title: 'DVC: Data Version Control - Git for Data & Models"'
+message: >-
+  If you use this software as part of a publication and wish
+  to cite it, please use the metadata from this file.
+type: software
+authors:
+  - name: The DVC team and contributors
+    website: 'https://github.com/treeverse'
+repository-code: 'https://github.com/treeverse/dvc'
+url: 'https://dvc.org'
+abstract: >-
+  DVC is a tool for data science that takes advantage of existing software engineering toolset.
+  It helps machine learning teams manage large datasets, make projects reproducible, and collaborate better.
+keywords:
+  - ai
+  - collaboration
+  - data-science
+  - data-version-control
+  - developer-tools
+  - git
+  - machine-learning
+  - python
+  - reproducibility
+license: Apache-2.0

LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2017-2021 Iterative, Inc.
+   Copyright 2025 Treeverse.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.rst

@@ -167,7 +167,7 @@ To install the development version, run:
 
 .. code-block:: bash
 
-   pip install git+git://github.com/iterative/dvc
+   pip install "dvc @ git+https://github.com/treeverse/dvc"
 
 Package (Platform-specific)
 ---------------------------
@@ -175,7 +175,7 @@ Package (Platform-specific)
 |Packages|
 
 Self-contained packages for Linux, Windows, and Mac are available.
-The latest version of the packages can be found on the GitHub `releases page <https://github.com/iterative/dvc/releases>`_.
+The latest version of the packages can be found on the GitHub `releases page <https://github.com/treeverse/dvc/releases>`_.
 
 Ubuntu / Debian (deb)
 ^^^^^^^^^^^^^^^^^^^^^
@@ -198,8 +198,6 @@ Fedora / CentOS (rpm)
 Contributing
 ============
 
-|Maintainability|
-
 Contributions are welcome!
 Please see our `Contributing Guide <https://dvc.org/doc/user-guide/contributing/core>`_ for more details.
 Thanks to all our contributors!
@@ -213,7 +211,7 @@ Community and Support
 * `Forum <https://discuss.dvc.org/>`_
 * `Discord Chat <https://dvc.org/chat>`_
 * `Email <mailto:[email protected]>`_
-* `Mailing List <https://sweedom.us10.list-manage.com/subscribe/post?u=a08bf93caae4063c4e6a351f6&id=24c0ecc49a>`_
+* `Mailing List <https://dvc.org/community#subscribe>`_
 
 Copyright
 =========
@@ -237,23 +235,19 @@ Barrak, A., Eghan, E.E. and Adams, B. `On the Co-evolution of ML Pipelines and S
    :target: https://dvc.org
    :alt: DVC logo
 
-.. |VS Code Extension Overview| image:: https://raw.githubusercontent.com/iterative/vscode-dvc/main/extension/docs/overview.gif
+.. |VS Code Extension Overview| image:: https://raw.githubusercontent.com/treeverse/vscode-dvc/main/extension/docs/overview.gif
    :alt: DVC Extension for VS Code
 
-.. |CI| image:: https://github.com/iterative/dvc/workflows/Tests/badge.svg?branch=main
-   :target: https://github.com/iterative/dvc/actions
+.. |CI| image:: https://github.com/treeverse/dvc/actions/workflows/tests.yaml/badge.svg
+   :target: https://github.com/treeverse/dvc/actions/workflows/tests.yaml
    :alt: GHA Tests
 
-.. |Maintainability| image:: https://codeclimate.com/github/iterative/dvc/badges/gpa.svg
-   :target: https://codeclimate.com/github/iterative/dvc
-   :alt: Code Climate
-
 .. |Python Version| image:: https://img.shields.io/pypi/pyversions/dvc
    :target: https://pypi.org/project/dvc
    :alt: Python Version
 
-.. |Coverage| image:: https://codecov.io/gh/iterative/dvc/branch/main/graph/badge.svg
-   :target: https://codecov.io/gh/iterative/dvc
+.. |Coverage| image:: https://codecov.io/gh/treeverse/dvc/branch/main/graph/badge.svg
+   :target: https://codecov.io/gh/treeverse/dvc
    :alt: Codecov
 
 .. |Snap| image:: https://img.shields.io/badge/snap-install-82BEA0.svg?logo=snapcraft
@@ -268,7 +262,7 @@ Barrak, A., Eghan, E.E. and Adams, B. `On the Co-evolution of ML Pipelines and S
    :target: https://formulae.brew.sh/formula/dvc
    :alt: Homebrew
 
-.. |Conda| image:: https://img.shields.io/conda/v/conda-forge/dvc.svg?label=conda&logo=conda-forge
+.. |Conda| image:: https://anaconda.org/conda-forge/dvc/badges/version.svg
    :target: https://anaconda.org/conda-forge/dvc
    :alt: Conda-forge
 
@@ -292,8 +286,8 @@ Barrak, A., Eghan, E.E. and Adams, B. `On the Co-evolution of ML Pipelines and S
    :target: https://dvc.org/img/flow.gif
    :alt: how_dvc_works
 
-.. |Contribs| image:: https://contrib.rocks/image?repo=iterative/dvc
-   :target: https://github.com/iterative/dvc/graphs/contributors
+.. |Contribs| image:: https://contrib.rocks/image?repo=treeverse/dvc
+   :target: https://github.com/treeverse/dvc/graphs/contributors
    :alt: Contributors
 
 .. |VS Code| image:: https://img.shields.io/visual-studio-marketplace/v/Iterative.dvc?color=blue&label=VSCode&logo=visualstudiocode&logoColor=blue

dvc/__init__.py

@@ -3,6 +3,7 @@ DVC
 ----
 Make your data science projects reproducible and shareable.
 """
+
 import dvc.logger
 from dvc.build import PKG  # noqa: F401
 from dvc.version import __version__, version_tuple  # noqa: F401

dvc/__main__.py

@@ -1,4 +1,5 @@
 """Main entry point for DVC command line tool."""
+
 import sys
 
 from dvc.cli import main

dvc/__pyinstaller/hook-celery.py

@@ -5,13 +5,10 @@ from PyInstaller.utils.hooks import collect_submodules, is_module_or_submodule
 # Celery dynamically imports most celery internals at runtime
 # pyinstaller hook must expose all modules loaded by
 # kombu.utils.imports:symbol_by_name()
-_EXCLUDES = (
-    "celery.bin",
-    "celery.contrib",
-)
+_EXCLUDES = ("celery.bin", "celery.contrib")
 hiddenimports = collect_submodules(
     "celery",
-    filter=lambda name: not any(
-        is_module_or_submodule(name, module) for module in _EXCLUDES
+    filter=lambda name: (
+        not any(is_module_or_submodule(name, module) for module in _EXCLUDES)
     ),
 )

dvc/__pyinstaller/hook-dvc.py

@@ -8,7 +8,6 @@ datas += copy_metadata("gcsfs")
 datas += copy_metadata("pyarrow")
 datas += copy_metadata("pydrive2")
 datas += copy_metadata("s3fs", recursive=True)
-datas += copy_metadata("boto3")
 datas += copy_metadata("ossfs")
 datas += copy_metadata("sshfs")
 datas += copy_metadata("webdav4")

dvc/_debug.py

@@ -14,7 +14,7 @@ def viztracer_profile(
     log_async: bool = False,
 ):
     try:
-        import viztracer
+        import viztracer  # ty: ignore[unresolved-import]
     except ImportError:
         print("Failed to run profiler, viztracer is not installed")  # noqa: T201
         yield
@@ -37,7 +37,7 @@ def yappi_profile(
     separate_threads: Optional[bool] = False,
 ):
     try:
-        import yappi
+        import yappi  # ty: ignore[unresolved-import]
     except ImportError:
         print("Failed to run profiler, yappi is not installed")  # noqa: T201
         yield
@@ -79,7 +79,7 @@ def yappi_profile(
 def instrument(html_output=False):
     """Run a statistical profiler"""
     try:
-        from pyinstrument import Profiler
+        from pyinstrument import Profiler  # ty: ignore[unresolved-import]
     except ImportError:
         print("Failed to run profiler, pyinstrument is not installed")  # noqa: T201
         yield
@@ -123,7 +123,7 @@ def debug():
         yield
     except Exception:
         try:
-            import ipdb as pdb  # noqa: T100
+            import ipdb as pdb  # noqa: T100  # ty: ignore[unresolved-import]
         except ImportError:
             import pdb  # type: ignore[no-redef]  # noqa: T100
         pdb.post_mortem()
@@ -162,7 +162,7 @@ def _get_path_func(tool: str, ext: str):
     fmt = f"{tool}.dvc-{{now:%Y%m%d}}_{{now:%H%M%S}}.{ext}"
 
     def func(now: Optional["datetime"] = None) -> str:
-        return fmt.format(now=now or datetime.now())
+        return fmt.format(now=now or datetime.now())  # noqa: DTZ005
 
     return func
 
@@ -206,7 +206,7 @@ def add_debugging_flags(parser):
     from argparse import SUPPRESS
 
     # For detailed info see:
-    # https://github.com/iterative/dvc/wiki/Debugging,-Profiling-and-Benchmarking-DVC
+    # https://github.com/treeverse/dvc/wiki/Debugging,-Profiling-and-Benchmarking-DVC
     args, _ = parser.parse_known_args()
     verbose = args.verbose
 

dvc/analytics.py

@@ -1,10 +1,14 @@
 import json
 import os
+from typing import TYPE_CHECKING, Optional
 
 from dvc.log import logger
 
 from .env import DVC_ANALYTICS_ENDPOINT, DVC_NO_ANALYTICS
 
+if TYPE_CHECKING:
+    from dvc.scm import Base
+
 logger = logger.getChild(__name__)
 
 
@@ -67,6 +71,7 @@ def send(path):
     thus, the need of removing such file afterwards.
     """
     import requests
+    from requests.exceptions import RequestException
 
     url = os.environ.get(DVC_ANALYTICS_ENDPOINT, "https://analytics.dvc.org")
     headers = {"content-type": "application/json"}
@@ -81,7 +86,7 @@ def send(path):
 
     try:
         requests.post(url, json=report, headers=headers, timeout=5)
-    except requests.exceptions.RequestException as e:
+    except RequestException as e:
         logger.trace("", exc_info=True)
         logger.debug("failed to send analytics report %s", str(e))
 
@@ -89,20 +94,62 @@ def send(path):
     os.remove(path)
 
 
-def _scm_in_use():
-    from dvc.exceptions import NotDvcRepoError
-    from dvc.repo import Repo
-    from dvc.scm import NoSCM
+def _git_remote_url(scm: Optional["Base"]) -> Optional[str]:
+    from dvc.scm import Git
 
-    from .scm import SCM, SCMError
+    if not isinstance(scm, Git):
+        return None
+
+    from dulwich.porcelain import get_remote_repo
+
+    dulwich_repo = scm.dulwich.repo
+    try:
+        _remote, url = get_remote_repo(dulwich_repo)
+    except IndexError:
+        # IndexError happens when the head is detached
+        _remote, url = get_remote_repo(dulwich_repo, b"origin")
+    # Dulwich returns (None, "origin") if no remote set
+    if (_remote, url) == (None, "origin"):
+        return None
+    return url
+
+
+def _scm_in_use(scm: Optional["Base"]) -> Optional[str]:
+    return type(scm).__name__ if scm else None
+
+
+def _parse_git_remote_path(remote_url: str) -> str:
+    from urllib.parse import urlparse
+
+    from scmrepo.urls import is_scp_style_url
+
+    parsed = urlparse(remote_url)
+    # Windows Path also gets parsed with a drive letter as scheme
+    # https://github.com/python/cpython/issues/86381
+    if parsed.scheme and parsed.scheme in ("http", "https", "git", "ssh"):
+        return parsed.path.strip("/")
+
+    if is_scp_style_url(remote_url):
+        # handle scp-style URL
+        parts = remote_url.split(":", 1)
+        if len(parts) == 2:
+            _, path = parts
+            return path.rstrip("/")
+    return remote_url
+
+
+def _git_remote_path_hash(scm: Optional["Base"]) -> Optional[str]:
+    """Return a hash of the git remote path."""
+    import hashlib
 
     try:
-        scm = SCM(root_dir=Repo.find_root())
-        return type(scm).__name__
-    except SCMError:
-        return NoSCM.__name__
-    except NotDvcRepoError:
-        pass
+        if remote_url := _git_remote_url(scm):
+            path = _parse_git_remote_path(remote_url)
+            h = hashlib.md5(path.encode("utf-8"), usedforsecurity=False)  # for FIPS
+            return h.hexdigest()
+    except Exception:
+        logger.debug("Failed to get git remote path", exc_info=True)
+    return None
 
 
 def _runtime_info():
@@ -112,6 +159,8 @@ def _runtime_info():
     from iterative_telemetry import _generate_ci_id, find_or_create_user_id
 
     from dvc import __version__
+    from dvc.info import _get_remotes
+    from dvc.repo import Repo
     from dvc.utils import is_binary
 
     ci_id = _generate_ci_id()
@@ -120,13 +169,24 @@ def _runtime_info():
     else:
         group_id, user_id = None, find_or_create_user_id()
 
+    scm = None
+    remotes = None
+    try:
+        repo = Repo()
+        scm = repo.scm
+        remotes = _get_remotes(repo.config)
+    except Exception as exc:  # noqa: BLE001
+        logger.debug("failed to open repo: %s", exc)
+
     return {
         "dvc_version": __version__,
         "is_binary": is_binary(),
-        "scm_class": _scm_in_use(),
+        "scm_class": _scm_in_use(scm),
         "system_info": _system_info(),
         "user_id": user_id,
         "group_id": group_id,
+        "remotes": remotes,
+        "git_remote_hash": _git_remote_path_hash(scm),
     }
 
 

dvc/annotations.py

@@ -1,5 +1,5 @@
 from dataclasses import asdict, dataclass, field, fields
-from typing import Any, ClassVar, Dict, List, Optional
+from typing import Any, ClassVar, Optional
 
 from funcy import compact
 from voluptuous import Required
@@ -13,11 +13,11 @@ class Annotation:
     PARAM_META: ClassVar[str] = "meta"
 
     desc: Optional[str] = None
-    type: Optional[str] = None  # noqa: A003
-    labels: List[str] = field(default_factory=list)
-    meta: Dict[str, Any] = field(default_factory=dict)
+    type: Optional[str] = None
+    labels: list[str] = field(default_factory=list)
+    meta: dict[str, Any] = field(default_factory=dict)
 
-    def to_dict(self) -> Dict[str, str]:
+    def to_dict(self) -> dict[str, str]:
         return compact(asdict(self))
 
 
@@ -31,11 +31,11 @@ class Artifact:
 
     path: str
     desc: Optional[str] = None
-    type: Optional[str] = None  # noqa: A003
-    labels: List[str] = field(default_factory=list)
-    meta: Dict[str, Any] = field(default_factory=dict)
+    type: Optional[str] = None
+    labels: list[str] = field(default_factory=list)
+    meta: dict[str, Any] = field(default_factory=dict)
 
-    def to_dict(self) -> Dict[str, str]:
+    def to_dict(self) -> dict[str, str]:
         return compact(asdict(self))
 
 
@@ -46,7 +46,7 @@ ANNOTATION_SCHEMA = {
     Annotation.PARAM_LABELS: [str],
     Annotation.PARAM_META: object,
 }
-ARTIFACT_SCHEMA: Dict[Any, Any] = {
+ARTIFACT_SCHEMA: dict[Any, Any] = {
     Required(Artifact.PARAM_PATH): str,
     **ANNOTATION_SCHEMA,  # type: ignore[arg-type]
 }

dvc/api/__init__.py

@@ -1,22 +1,24 @@
 from dvc.fs.dvc import _DVCFileSystem as DVCFileSystem
 
 from .artifacts import artifacts_show
-from .data import get_url, open, read
+from .data import get_url, open, read  # noqa: A004
+from .dataset import get as get_dataset
 from .experiments import exp_save, exp_show
 from .scm import all_branches, all_commits, all_tags
 from .show import metrics_show, params_show
 
 __all__ = [
+    "DVCFileSystem",
     "all_branches",
     "all_commits",
     "all_tags",
     "artifacts_show",
     "exp_save",
     "exp_show",
+    "get_dataset",
     "get_url",
+    "metrics_show",
     "open",
     "params_show",
-    "metrics_show",
     "read",
-    "DVCFileSystem",
 ]

dvc/api/artifacts.py

@@ -1,4 +1,5 @@
-from typing import Any, Dict, Optional
+import os
+from typing import Any, Optional
 
 from dvc.repo import Repo
 
@@ -8,7 +9,7 @@ def artifacts_show(
     version: Optional[str] = None,
     stage: Optional[str] = None,
     repo: Optional[str] = None,
-) -> Dict[str, str]:
+) -> dict[str, str]:
     """
     Return path and Git revision for an artifact in a DVC project.
 
@@ -36,12 +37,22 @@ def artifacts_show(
     if version and stage:
         raise ValueError("Artifact version and stage are mutually exclusive.")
 
-    repo_kwargs: Dict[str, Any] = {
+    from dvc.repo.artifacts import Artifacts
+    from dvc.utils import as_posix
+
+    repo_kwargs: dict[str, Any] = {
         "subrepos": True,
         "uninitialized": True,
     }
+
+    dirname, _ = Artifacts.parse_path(name)
     with Repo.open(repo, **repo_kwargs) as _repo:
         rev = _repo.artifacts.get_rev(name, version=version, stage=stage)
         with _repo.switch(rev):
-            path = _repo.artifacts.get_path(name)
-        return {"rev": rev, "path": path}
+            root = _repo.fs.root_marker
+            _dirname = _repo.fs.join(root, dirname) if dirname else root
+            with Repo(_dirname, fs=_repo.fs, scm=_repo.scm) as r:
+                path = r.artifacts.get_path(name)
+                path = _repo.fs.join(_repo.fs.root_marker, as_posix(path))
+                parts = _repo.fs.relparts(path, _repo.root_dir)
+                return {"rev": rev, "path": os.path.join(*parts)}

dvc/api/data.py

@@ -1,6 +1,6 @@
 from contextlib import _GeneratorContextManager as GCM
 from contextlib import contextmanager
-from typing import Any, Dict, Optional
+from typing import Any, Optional
 
 from funcy import reraise
 
@@ -25,7 +25,14 @@ def _wrap_exceptions(repo, url):
         raise PathMissingError(exc.path, url) from exc
 
 
-def get_url(path, repo=None, rev=None, remote=None):
+def get_url(
+    path: str,
+    repo: Optional[str] = None,
+    rev: Optional[str] = None,
+    remote: Optional[str] = None,
+    config: Optional[dict[str, Any]] = None,
+    remote_config: Optional[dict[str, Any]] = None,
+):
     """
     Returns the URL to the storage location of a data file or directory tracked
     in a DVC repo. For Git repos, HEAD is used unless a rev argument is
@@ -35,15 +42,44 @@ def get_url(path, repo=None, rev=None, remote=None):
 
     NOTE: This function does not check for the actual existence of the file or
     directory in the remote storage.
+
+    Args:
+        path (str): location and file name of the target, relative to the root
+            of `repo`.
+        repo (str, optional): location of the DVC project or Git Repo.
+            Defaults to the current DVC project (found by walking up from the
+            current working directory tree).
+            It can be a URL or a file system path.
+            Both HTTP and SSH protocols are supported for online Git repos
+            (e.g. [user@]server:project.git).
+        rev (str, optional): Any `Git revision`_ such as a branch or tag name,
+            a commit hash or a dvc experiment name.
+            Defaults to HEAD.
+            If `repo` is not a Git repo, this option is ignored.
+        remote (str, optional): Name of the `DVC remote`_ used to form the
+            returned URL string.
+            Defaults to the `default remote`_ of `repo`.
+            For local projects, the cache is tried before the default remote.
+        config(dict, optional): config to be passed to the DVC repository.
+            Defaults to None.
+        remote_config(dict, optional): remote config to be passed to the DVC
+            repository.
+            Defaults to None.
+
+    Returns:
+        str: URL to the file or directory.
     """
     from dvc.config import NoRemoteError
     from dvc_data.index import StorageKeyError
 
-    repo_kwargs: Dict[str, Any] = {}
-    if remote:
-        repo_kwargs["config"] = {"core": {"remote": remote}}
     with Repo.open(
-        repo, rev=rev, subrepos=True, uninitialized=True, **repo_kwargs
+        repo,
+        rev=rev,
+        subrepos=True,
+        uninitialized=True,
+        remote=remote,
+        config=config,
+        remote_config=remote_config,
     ) as _repo:
         index, entry = _repo.get_data_index_entry(path)
         with reraise(
@@ -57,11 +93,7 @@ def get_url(path, repo=None, rev=None, remote=None):
 class _OpenContextManager(GCM):
     def __init__(self, func, args, kwds):
         self.gen = func(*args, **kwds)
-        self.func, self.args, self.kwds = (  # type: ignore[assignment]
-            func,
-            args,
-            kwds,
-        )
+        self.func, self.args, self.kwds = (func, args, kwds)  # type: ignore[assignment]
 
     def __getattr__(self, name):
         raise AttributeError("dvc.api.open() should be used in a with statement.")
@@ -74,8 +106,8 @@ def open(  # noqa: A001
     remote: Optional[str] = None,
     mode: str = "r",
     encoding: Optional[str] = None,
-    config: Optional[Dict[str, Any]] = None,
-    remote_config: Optional[Dict[str, Any]] = None,
+    config: Optional[dict[str, Any]] = None,
+    remote_config: Optional[dict[str, Any]] = None,
 ):
     """
     Opens a file tracked in a DVC project.
@@ -233,7 +265,7 @@ def _open(
     config=None,
     remote_config=None,
 ):
-    repo_kwargs: Dict[str, Any] = {
+    repo_kwargs: dict[str, Any] = {
         "subrepos": True,
         "uninitialized": True,
         "remote": remote,
@@ -253,7 +285,7 @@ def _open(
             if TYPE_CHECKING:
                 from dvc.fs import FileSystem
 
-            fs: Union["FileSystem", DataFileSystem, DVCFileSystem]
+            fs: Union[FileSystem, DataFileSystem, DVCFileSystem]
             if os.path.isabs(path):
                 fs = DataFileSystem(index=_repo.index.data["local"])
                 fs_path = path
@@ -262,11 +294,7 @@ def _open(
                 fs_path = fs.from_os_path(path)
 
             try:
-                with fs.open(
-                    fs_path,
-                    mode=mode,
-                    encoding=encoding,
-                ) as fobj:
+                with fs.open(fs_path, mode=mode, encoding=encoding) as fobj:
                     yield fobj
             except FileNotFoundError as exc:
                 raise FileMissingError(path) from exc

dvc/api/dataset.py

@@ -0,0 +1,67 @@
+from typing import Literal, TypedDict, Union
+
+
+class DatachainDataset(TypedDict):
+    type: Literal["dc"]
+    name: str
+    version: int
+
+
+class DVCDataset(TypedDict):
+    type: Literal["dvc"]
+    url: str
+    path: str
+    sha: str
+
+
+class URLDataset(TypedDict):
+    type: Literal["url"]
+    files: list[str]
+    path: str
+
+
+def get(name: str) -> Union[DatachainDataset, DVCDataset, URLDataset]:
+    from difflib import get_close_matches
+
+    from dvc.fs import get_cloud_fs
+    from dvc.repo import Repo, datasets
+
+    repo = Repo()
+    try:
+        dataset = repo.datasets[name]
+    except datasets.DatasetNotFoundError as e:
+        add_note = getattr(e, "add_note", lambda _: None)
+        if matches := get_close_matches(name, repo.datasets):
+            add_note(f"Did you mean: {matches[0]!r}?")
+        raise
+
+    if dataset._invalidated:
+        raise ValueError(f"dataset not in sync. Sync with 'dvc ds update {name}'.")
+    if not dataset.lock:
+        raise ValueError("missing lock information")
+    if dataset.type == "dvc":
+        return DVCDataset(
+            type="dvc",
+            url=dataset.lock.url,
+            path=dataset.lock.path,
+            sha=dataset.lock.rev_lock,
+        )
+    if dataset.type == "dc":
+        return DatachainDataset(
+            type="dc", name=dataset.name_version[0], version=dataset.lock.version
+        )
+    if dataset.type == "url":
+        fs_cls, _, path = get_cloud_fs(repo.config, url=dataset.lock.url)
+        assert fs_cls
+        join_version = getattr(fs_cls, "join_version", lambda path, _: path)
+        protocol = fs_cls.protocol
+        versioned_path = join_version(path, dataset.lock.meta.version_id)
+        versioned_path = f"{protocol}://{versioned_path}"
+        files = [
+            join_version(
+                fs_cls.join(versioned_path, file.relpath), file.meta.version_id
+            )
+            for file in dataset.lock.files
+        ]
+        return URLDataset(type="url", files=files, path=versioned_path)
+    raise AssertionError("unreachable")

dvc/api/experiments.py

@@ -1,4 +1,4 @@
-from typing import Dict, List, Optional, Union
+from typing import Optional, Union
 
 from rich.text import Text
 
@@ -9,7 +9,7 @@ from dvc.repo.experiments.show import tabulate
 def exp_save(
     name: Optional[str] = None,
     force: bool = False,
-    include_untracked: Optional[List[str]] = None,
+    include_untracked: Optional[list[str]] = None,
 ):
     """
     Create a new DVC experiment using `exp save`.
@@ -61,12 +61,12 @@ def _postprocess(exp_rows):
 
 def exp_show(
     repo: Optional[str] = None,
-    revs: Optional[Union[str, List[str]]] = None,
+    revs: Optional[Union[str, list[str]]] = None,
     num: int = 1,
     param_deps: bool = False,
     force: bool = False,
-    config: Optional[Dict] = None,
-) -> List[Dict]:
+    config: Optional[dict] = None,
+) -> list[dict]:
     """Get DVC experiments tracked in `repo`.
 
     Without arguments, this function will retrieve all experiments derived from

dvc/api/scm.py

@@ -1,9 +1,9 @@
-from typing import List, Optional
+from typing import Optional
 
 from dvc.repo import Repo
 
 
-def all_branches(repo: Optional[str] = None) -> List[str]:
+def all_branches(repo: Optional[str] = None) -> list[str]:
     """Get all Git branches in a DVC repository.
 
     Args:
@@ -20,7 +20,7 @@ def all_branches(repo: Optional[str] = None) -> List[str]:
         return _repo.scm.list_branches()
 
 
-def all_commits(repo: Optional[str] = None) -> List[str]:
+def all_commits(repo: Optional[str] = None) -> list[str]:
     """Get all Git commits in a DVC repository.
 
     Args:
@@ -37,7 +37,7 @@ def all_commits(repo: Optional[str] = None) -> List[str]:
         return _repo.scm.list_all_commits()
 
 
-def all_tags(repo: Optional[str] = None) -> List[str]:
+def all_tags(repo: Optional[str] = None) -> list[str]:
     """Get all Git tags in a DVC repository.
 
     Args:

dvc/api/show.py

@@ -1,6 +1,7 @@
 import typing
 from collections import Counter
-from typing import Dict, Iterable, Optional, Union
+from collections.abc import Iterable
+from typing import Optional, Union
 
 from funcy import first
 
@@ -8,7 +9,7 @@ from dvc.repo import Repo
 
 
 def _postprocess(results):
-    processed: Dict[str, Dict] = {}
+    processed: dict[str, dict] = {}
     for rev, rev_data in results.items():
         if not rev_data:
             continue
@@ -24,10 +25,9 @@ def _postprocess(results):
                 (k if counts[k] == 1 else f"{file_name}:{k}"): v
                 for k, v in file_data["data"].items()
             }
-            processed[rev] = {**processed[rev], **to_merge}
+            processed[rev] = processed[rev] | to_merge
 
-    if "workspace" in processed:
-        del processed["workspace"]
+    processed.pop("workspace", None)
 
     return processed
 
@@ -36,8 +36,8 @@ def metrics_show(
     *targets: str,
     repo: Optional[str] = None,
     rev: Optional[str] = None,
-    config: Optional[Dict] = None,
-) -> Dict:
+    config: Optional[dict] = None,
+) -> dict:
     """Get metrics tracked in `repo`.
 
     Without arguments, this function will retrieve all metrics from all tracked
@@ -159,8 +159,8 @@ def params_show(
     stages: Optional[Union[str, Iterable[str]]] = None,
     rev: Optional[str] = None,
     deps: bool = False,
-    config: Optional[Dict] = None,
-) -> Dict:
+    config: Optional[dict] = None,
+) -> dict:
     """Get parameters tracked in `repo`.
 
     Without arguments, this function will retrieve all params from all tracked

dvc/cachemgr.py

@@ -1,5 +1,5 @@
 import os
-from typing import TYPE_CHECKING, Optional, Tuple
+from typing import TYPE_CHECKING, Optional
 
 from dvc.fs import GitFileSystem, Schemes
 from dvc_data.hashfile.db import get_odb
@@ -15,7 +15,7 @@ def _get_odb(
     repo,
     settings,
     fs=None,
-    prefix: Optional[Tuple[str, ...]] = None,
+    prefix: Optional[tuple[str, ...]] = None,
     hash_name: Optional[str] = None,
     **kwargs,
 ):
@@ -86,7 +86,7 @@ class CacheManager:
         return FileStorage(
             key=(),
             fs=self.local.fs,
-            path=self.local.fs.join(self.default_local_cache_dir, self.FS_DIR),
+            path=self.local.fs.join(self.local_cache_dir, self.FS_DIR),
         )
 
     def _init_odb(self, schemes):
@@ -144,15 +144,9 @@ def migrate_2_to_3(repo: "Repo", dry: bool = False):
         )
         return
 
-    with TqdmCallback(
-        desc="Computing DVC 3.0 hashes",
-        unit="files",
-    ) as cb:
+    with TqdmCallback(desc="Computing DVC 3.0 hashes", unit="files") as cb:
         migration = prepare(src, dest, callback=cb)
 
-    with TqdmCallback(
-        desc="Migrating to DVC 3.0 cache",
-        unit="files",
-    ) as cb:
+    with TqdmCallback(desc="Migrating to DVC 3.0 cache", unit="files") as cb:
         count = migrate(migration, callback=cb)
     ui.write(f"Migrated {count} files to DVC 3.0 cache location.")

dvc/cli/__init__.py

@@ -96,7 +96,7 @@ def _log_exceptions(exc: Exception) -> Optional[int]:
                 f"See {link} for more info."
             )
         else:
-            link = format_link("https://github.com/iterative/dvc/issues")
+            link = format_link("https://github.com/treeverse/dvc/issues")
             hint = f"\nPlease report this bug to {link}. Thank you!"
 
         logger.exception(
@@ -157,6 +157,8 @@ def main(argv=None):  # noqa: C901, PLR0912, PLR0915
     Returns:
         int: command's return code.
     """
+    from contextlib import ExitStack
+
     from dvc._debug import debugtools
     from dvc.config import ConfigError
     from dvc.exceptions import DvcException, NotDvcRepoError
@@ -173,12 +175,11 @@ def main(argv=None):  # noqa: C901, PLR0912, PLR0915
         logging.disable(logging.INFO)
 
     args = None
-
-    outer_log_level = logger.level
-    level = None
+    stack = ExitStack()
     try:
         args = parse_args(argv)
 
+        level = None
         if args.quiet:
             level = logging.CRITICAL
         elif args.verbose == 1:
@@ -187,7 +188,7 @@ def main(argv=None):  # noqa: C901, PLR0912, PLR0915
             level = logging.TRACE  # type: ignore[attr-defined]
 
         if level is not None:
-            set_loggers_level(level)
+            stack.enter_context(set_loggers_level(level))
 
         if level and level <= logging.DEBUG:
             from platform import platform, python_implementation, python_version
@@ -247,7 +248,7 @@ def main(argv=None):  # noqa: C901, PLR0912, PLR0915
 
         return ret
     finally:
-        logger.setLevel(outer_log_level)
+        stack.close()
 
         from dvc.repo.open_repo import clean_repos
 

dvc/cli/command.py

@@ -18,8 +18,11 @@ class CmdBase(ABC):
 
         os.chdir(args.cd)
 
-        self.repo: "Repo" = Repo(uninitialized=self.UNINITIALIZED)
-        self.config: "Config" = self.repo.config
+        self.repo: Repo = Repo(
+            uninitialized=self.UNINITIALIZED,
+            _wait_for_lock=args.wait_for_lock,
+        )
+        self.config: Config = self.repo.config
         self.args = args
 
     def do_run(self):

dvc/cli/completion.py

@@ -22,6 +22,15 @@ _dvc_compgen_exps() {
     local _dvc_exps=($(dvc exp list -q --all-commits --names-only))
     compgen -W "${_dvc_exps[*]}" -- $1
 }
+
+_dvc_compgen_remotes() {
+    local _dvc_remotes=($(dvc remote list | cut -d' ' -f1))
+    compgen -W "${_dvc_remotes[*]}" -- $1
+}
+
+_dvc_compgen_config_vars() {
+    compgen -W "${_dvc_config_vars[*]}" -- $1
+}
 """
 
 ZSH_PREAMBLE = """
@@ -55,6 +64,14 @@ _dvc_compadd_stages_and_files() {
 _dvc_compadd_exps() {
     _describe 'experiments' "($(dvc exp list -q --all-commits --names-only))"
 }
+
+_dvc_compadd_remotes() {
+    _describe 'remotes' "($(dvc remote list | cut -d' ' -f1))"
+}
+
+_dvc_compadd_config_vars() {
+    _describe 'config_vars' _dvc_config_vars
+}
 """
 
 PREAMBLE = {
@@ -71,3 +88,22 @@ DVCFILES_AND_STAGE = {
     "zsh": "_dvc_compadd_stages_and_files",
 }
 EXPERIMENT = {"bash": "_dvc_compgen_exps", "zsh": "_dvc_compadd_exps"}
+REMOTE = {"bash": "_dvc_compgen_remotes", "zsh": "_dvc_compadd_remotes"}
+CONFIG_VARS = {"bash": "_dvc_compgen_config_vars", "zsh": "_dvc_compadd_config_vars"}
+
+
+def get_preamble() -> dict[str, str]:
+    from dvc.config_schema import config_vars_for_completion
+
+    ret: dict[str, str] = {}
+    config_vars = list(config_vars_for_completion())
+
+    nl = "\n\t".expandtabs(4)
+    config_vars_arr = f"""
+_dvc_config_vars=(
+    {nl.join(config_vars)}
+)
+"""
+    for shell, preamble in PREAMBLE.items():
+        ret[shell] = config_vars_arr + preamble
+    return ret

dvc/cli/parser.py

@@ -1,4 +1,5 @@
 """Main parser for the dvc cli."""
+
 import argparse
 import os
 from functools import lru_cache
@@ -17,6 +18,7 @@ from dvc.commands import (
     dag,
     data,
     data_sync,
+    dataset,
     destroy,
     diff,
     du,
@@ -55,48 +57,49 @@ from . import DvcParserError, formatter
 logger = logger.getChild(__name__)
 
 COMMANDS = [
-    init,
-    queue,
+    add,
+    artifacts,
+    cache,
+    check_ignore,
+    checkout,
+    commit,
+    completion,
+    config,
+    daemon,
+    dag,
+    data,
+    data_sync,
+    dataset,
+    destroy,
+    diff,
+    du,
+    experiments,
+    freeze,
+    gc,
     get,
     get_url,
-    destroy,
-    add,
-    remove,
-    move,
-    unprotect,
-    repro,
-    data_sync,
-    gc,
+    git_hook,
     imp,
     imp_db,
     imp_url,
-    config,
-    checkout,
-    remote,
-    cache,
-    metrics,
-    params,
+    init,
     install,
-    root,
     ls,
     ls_url,
-    freeze,
-    dag,
-    daemon,
-    commit,
-    completion,
-    diff,
-    version,
-    update,
-    git_hook,
+    metrics,
+    move,
+    params,
     plots,
+    queue,
+    remote,
+    remove,
+    repro,
+    root,
     stage,
-    experiments,
-    check_ignore,
-    data,
-    artifacts,
     studio,
-    du,
+    unprotect,
+    update,
+    version,
 ]
 
 
@@ -197,6 +200,14 @@ def get_main_parser():
         type=str,
     )
 
+    parser.add_argument(
+        "--wait-for-lock",
+        action="store_true",
+        default=False,
+        help="Wait for the lock if it is already held by another process, instead of"
+        " failing immediately.",
+    )
+
     # Sub commands
     subparsers = parser.add_subparsers(
         title="Available Commands",

dvc/cli/utils.py

@@ -40,7 +40,7 @@ def hide_subparsers_from_help(subparsers):
     # from the 'positional arguments' choices list
     # see: https://bugs.python.org/issue22848
     # Need to set `add_help=False`, but avoid setting `help`
-    # (not even to `argparse.SUPPPRESS`).
+    # (not even to `argparse.SUPPRESS`).
     # NOTE: The argument is the parent subparser, not the subcommand parser.
     cmds = [cmd for cmd, parser in subparsers.choices.items() if parser.add_help]
     subparsers.metavar = "{{{}}}".format(",".join(cmds))

dvc/commands/add.py

@@ -51,6 +51,7 @@ class CmdAdd(CmdBase):
                 to_remote=self.args.to_remote,
                 remote_jobs=self.args.remote_jobs,
                 force=self.args.force,
+                relink=self.args.relink,
             )
         except FileNotFoundError:
             logger.exception("")
@@ -100,7 +101,7 @@ def add_parser(subparsers, parent_parser):
         "--remote",
         help="Remote storage to download to",
         metavar="<name>",
-    )
+    ).complete = completion.REMOTE
     parser.add_argument(
         "--remote-jobs",
         type=int,
@@ -119,6 +120,13 @@ def add_parser(subparsers, parent_parser):
         default=False,
         help="Override local file or folder if exists.",
     )
+    parser.add_argument(
+        "--no-relink",
+        dest="relink",
+        action="store_false",
+        help="Don't recreate links from cache to workspace.",
+    )
+    parser.set_defaults(relink=True)
     parser.add_argument(
         "targets", nargs="+", help="Input files/directories to add."
     ).complete = completion.FILE

dvc/commands/artifacts.py

@@ -13,6 +13,9 @@ class CmdArtifactsGet(CmdBaseNoRepo):
         from dvc.scm import CloneError
         from dvc.ui import ui
 
+        if self.args.show_url:
+            return self._show_url()
+
         try:
             count, out = Artifacts.get(
                 self.args.url,
@@ -36,6 +39,28 @@ class CmdArtifactsGet(CmdBaseNoRepo):
             )
             return 1
 
+    def _show_url(self):
+        from dvc.api import artifacts_show, get_url
+        from dvc.ui import ui
+
+        artifact = artifacts_show(
+            self.args.name,
+            version=self.args.rev,
+            stage=self.args.stage,
+            repo=self.args.url,
+        )
+
+        url = get_url(
+            artifact["path"],
+            repo=self.args.url,
+            rev=artifact["rev"],
+            remote=self.args.remote,
+            remote_config=self.args.remote_config,
+        )
+        ui.write(url, force=True)
+
+        return 0
+
 
 def add_parser(subparsers, parent_parser):
     ARTIFACTS_HELP = "DVC model registry artifact commands."
@@ -84,6 +109,14 @@ def add_parser(subparsers, parent_parser):
         help="Destination path to download artifact to",
         metavar="<path>",
     ).complete = completion.DIR
+    get_parser.add_argument(
+        "--show-url",
+        action="store_true",
+        help=(
+            "Print the storage location (URL) the target data would be "
+            "downloaded from, and exit."
+        ),
+    )
     get_parser.add_argument(
         "-j",
         "--jobs",
@@ -116,7 +149,7 @@ def add_parser(subparsers, parent_parser):
             "Remote name to set as a default in the target repository "
             "(only applicable when downloading from DVC remote)."
         ),
-    )
+    ).complete = completion.REMOTE
     get_parser.add_argument(
         "--remote-config",
         type=str,

dvc/commands/check_ignore.py

@@ -1,20 +1,28 @@
+from typing import TYPE_CHECKING
+
 from dvc.cli import completion, formatter
 from dvc.cli.command import CmdBase
 from dvc.cli.utils import append_doc_link
 from dvc.ui import ui
 
+if TYPE_CHECKING:
+    from dvc.ignore import CheckIgnoreResult
+
 
 class CmdCheckIgnore(CmdBase):
     def __init__(self, args):
         super().__init__(args)
         self.ignore_filter = self.repo.dvcignore
 
-    def _show_results(self, result):
-        if not result.match and not self.args.non_matching:
-            return
+    def _show_results(self, result: "CheckIgnoreResult"):
+        if not result.match and not self.args.details:
+            return None
 
         if self.args.details:
-            patterns = result.patterns
+            pattern_infos = result.pattern_infos
+            patterns = [str(pi) for pi in pattern_infos]
+            if not patterns and self.args.non_matching:
+                patterns = ["::"]
             if not self.args.all:
                 patterns = patterns[-1:]
 
@@ -22,13 +30,11 @@ class CmdCheckIgnore(CmdBase):
                 ui.write(pattern, result.file, sep="\t")
         else:
             ui.write(result.file)
+        return bool(result.pattern_infos)
 
     def _check_one_file(self, target):
         result = self.ignore_filter.check_ignore(target)
-        self._show_results(result)
-        if result.match:
-            return 0
-        return 1
+        return bool(self._show_results(result))
 
     def _interactive_mode(self):
         ret = 1
@@ -39,14 +45,14 @@ class CmdCheckIgnore(CmdBase):
                 break
             if not target:
                 break
-            if not self._check_one_file(target):
+            if self._check_one_file(target):
                 ret = 0
         return ret
 
     def _normal_mode(self):
         ret = 1
         for target in self.args.targets:
-            if not self._check_one_file(target):
+            if self._check_one_file(target):
                 ret = 0
         return ret
 
@@ -76,6 +82,8 @@ class CmdCheckIgnore(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
+    import argparse
+
     ADD_HELP = "Check whether files or directories are excluded due to `.dvcignore`."
 
     parser = subparsers.add_parser(
@@ -93,14 +101,7 @@ def add_parser(subparsers, parent_parser):
         help="Show the exclude patterns along with each target path.",
     )
     parser.add_argument(
-        "-a",
-        "--all",
-        action="store_true",
-        default=False,
-        help=(
-            "Include the target paths which don't match any pattern "
-            "in the `--details` list."
-        ),
+        "-a", "--all", action="store_true", default=False, help=argparse.SUPPRESS
     )
     parser.add_argument(
         "-n",

dvc/commands/checkout.py

@@ -1,5 +1,3 @@
-import operator
-
 from dvc.cli import completion, formatter
 from dvc.cli.command import CmdBase
 from dvc.cli.utils import append_doc_link
@@ -30,7 +28,7 @@ class CmdCheckout(CmdBase):
 
         stats, exc = None, None
         try:
-            stats = self.repo.checkout(
+            result = self.repo.checkout(
                 targets=self.args.targets,
                 with_deps=self.args.with_deps,
                 force=self.args.force,
@@ -40,14 +38,17 @@ class CmdCheckout(CmdBase):
             )
         except CheckoutError as _exc:
             exc = _exc
-            stats = exc.stats
+            result = exc.result
 
         if self.args.summary:
             default_message = "No changes."
-            msg = get_summary(sorted(stats.items(), key=operator.itemgetter(0)))
+            stats = result["stats"]
+            assert isinstance(stats, dict)
+            msg = get_summary(stats.items())
             ui.write(msg or default_message)
         else:
-            log_changes(stats)
+            result.pop("stats", {})
+            log_changes(result)
 
         if exc:
             raise exc

dvc/commands/commit.py

@@ -20,6 +20,7 @@ class CmdCommit(CmdBase):
                     with_deps=self.args.with_deps,
                     recursive=self.args.recursive,
                     force=self.args.force,
+                    relink=self.args.relink,
                 )
             except DvcException:
                 logger.exception("failed to commit%s", (" " + target) if target else "")
@@ -64,6 +65,13 @@ def add_parser(subparsers, parent_parser):
         default=False,
         help="Commit cache for subdirectories of the specified directory.",
     )
+    commit_parser.add_argument(
+        "--no-relink",
+        dest="relink",
+        action="store_false",
+        help="Don't recreate links from cache to workspace.",
+    )
+    commit_parser.set_defaults(relink=True)
     commit_parser.add_argument(
         "targets",
         nargs="*",

dvc/commands/completion.py

@@ -1,6 +1,6 @@
 from dvc.cli import formatter
 from dvc.cli.command import CmdBaseNoRepo
-from dvc.cli.completion import PREAMBLE
+from dvc.cli.completion import get_preamble
 from dvc.cli.utils import append_doc_link
 from dvc.log import logger
 from dvc.ui import ui
@@ -17,7 +17,7 @@ class CmdCompletion(CmdBaseNoRepo):
 
         shell = self.args.shell
         parser = self.args.parser
-        script = shtab.complete(parser, shell=shell, preamble=PREAMBLE)
+        script = shtab.complete(parser, shell=shell, preamble=get_preamble())
         ui.write(script, force=True)
         return 0
 

dvc/commands/config.py

@@ -3,7 +3,7 @@ import os
 
 from funcy import set_in
 
-from dvc.cli import formatter
+from dvc.cli import completion, formatter
 from dvc.cli.command import CmdBaseNoRepo
 from dvc.cli.utils import append_doc_link
 from dvc.log import logger
@@ -224,7 +224,7 @@ def add_parser(subparsers, parent_parser):
         nargs="?",
         type=_name_type,
         help="Option name (section.option or remote.name.option).",
-    )
+    ).complete = completion.CONFIG_VARS
     config_parser.add_argument("value", nargs="?", help="Option value.")
     config_parser.add_argument(
         "-l",

dvc/commands/dag.py

@@ -17,7 +17,7 @@ def _show_ascii(graph: "DiGraph"):
 
     ret = []
     for pipeline in pipelines:
-        ret.append(draw(pipeline.nodes, pipeline.edges))
+        ret.append(draw(pipeline.nodes, pipeline.edges))  # noqa: PERF401
 
     return "\n".join(ret)
 
@@ -127,19 +127,80 @@ def _filter(graph, targets, full):
     return new_graph
 
 
-def _build(repo, target=None, full=False, outs=False):
+def _is_foreach_matrix_stage(node, join_string):
+    if node.endswith(".dvc"):
+        return False
+    return join_string in node
+
+
+def _collapse_foreach_matrix_get_nodes(graph):
+    from dvc.parsing import JOIN
+
+    new_nodes = set()
+    nodes_to_remove = set()
+    for _node in list(graph.nodes):
+        if not _is_foreach_matrix_stage(_node, JOIN):
+            continue
+        nodes_to_remove.add(_node)
+        new_nodes.add(_node.split(JOIN)[0])
+    return new_nodes, nodes_to_remove
+
+
+def _collapse_foreach_matrix_get_edges(graph):
+    from dvc.parsing import JOIN
+
+    new_edges = set()
+    edges_to_remove = set()
+    for _e1, _e2 in list(graph.edges):
+        _replace = False
+        _new_e1 = _e1
+        _new_e2 = _e2
+        if _is_foreach_matrix_stage(_e1, JOIN):
+            _new_e1 = _e1.split(JOIN)[0]
+            _replace = True
+        if _is_foreach_matrix_stage(_e2, JOIN):
+            _new_e2 = _e2.split(JOIN)[0]
+            _replace = True
+        if _replace:
+            edges_to_remove.add((_e1, _e2))
+            new_edges.add((_new_e1, _new_e2))
+    return new_edges, edges_to_remove
+
+
+def _collapse_foreach_matrix(graph):
+    new_nodes, nodes_to_remove = _collapse_foreach_matrix_get_nodes(graph)
+    new_edges, edges_to_remove = _collapse_foreach_matrix_get_edges(graph)
+    new_graph = graph.copy()
+    new_graph.remove_edges_from(edges_to_remove)
+    new_graph.add_nodes_from(new_nodes)
+    new_graph.add_edges_from(new_edges)
+    new_graph.remove_nodes_from(nodes_to_remove)
+    return new_graph
+
+
+def _build(repo, target=None, full=False, outs=False, collapse_foreach_matrix=False):
     targets = _collect_targets(repo, target, outs)
     graph = _transform(repo.index, outs)
-    return _filter(graph, targets, full)
+    filtered_graph = _filter(graph, targets, full)
+    if collapse_foreach_matrix:
+        filtered_graph = _collapse_foreach_matrix(filtered_graph)
+    return filtered_graph
 
 
 class CmdDAG(CmdBase):
     def run(self):
+        from dvc.exceptions import InvalidArgumentError
+
+        if self.args.outs and self.args.collapse_foreach_matrix:
+            raise InvalidArgumentError(
+                "`--outs` and `--collapse-foreach-matrix` are mutually exclusive"
+            )
         graph = _build(
             self.repo,
             target=self.args.target,
             full=self.args.full,
             outs=self.args.outs,
+            collapse_foreach_matrix=self.args.collapse_foreach_matrix,
         )
 
         if self.args.dot:
@@ -181,6 +242,14 @@ def add_parser(subparsers, parent_parser):
         dest="markdown",
         help="Print DAG with mermaid format wrapped in Markdown block.",
     )
+    dag_parser.add_argument(
+        "--collapse-foreach-matrix",
+        action="store_true",
+        default=False,
+        help=(
+            "Collapse stages from each foreach/matrix definition into a single node."
+        ),
+    )
     dag_parser.add_argument(
         "--full",
         action="store_true",

dvc/commands/data.py

@@ -1,8 +1,8 @@
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, ClassVar
 
 from funcy import chunks, compact, log_durations
 
-from dvc.cli import formatter
+from dvc.cli import completion, formatter
 from dvc.cli.command import CmdBase
 from dvc.cli.utils import append_doc_link
 from dvc.log import logger
@@ -10,6 +10,7 @@ from dvc.ui import ui
 from dvc.utils import colorize
 
 if TYPE_CHECKING:
+    from dvc.repo.data import GitInfo
     from dvc.repo.data import Status as DataStatus
 
 
@@ -17,14 +18,14 @@ logger = logger.getChild(__name__)
 
 
 class CmdDataStatus(CmdBase):
-    COLORS = {
+    COLORS: ClassVar[dict[str, str]] = {
         "not_in_remote": "red",
         "not_in_cache": "red",
         "committed": "green",
         "uncommitted": "yellow",
         "untracked": "cyan",
     }
-    LABELS = {
+    LABELS: ClassVar[dict[str, str]] = {
         "not_in_remote": "Not in remote",
         "not_in_cache": "Not in cache",
         "committed": "DVC committed changes",
@@ -32,7 +33,7 @@ class CmdDataStatus(CmdBase):
         "untracked": "Untracked files",
         "unchanged": "DVC unchanged files",
     }
-    HINTS = {
+    HINTS: ClassVar[dict[str, tuple[str, ...]]] = {
         "not_in_remote": ('use "dvc push <file>..." to upload files',),
         "not_in_cache": ('use "dvc fetch <file>..." to download files',),
         "committed": ("git commit the corresponding dvc files to update the repo",),
@@ -55,20 +56,15 @@ class CmdDataStatus(CmdBase):
     def _process_status(status: "DataStatus"):
         """Flatten stage status, and filter empty stage status contents."""
         for stage, stage_status in status.items():
-            items = stage_status
-            if isinstance(stage_status, dict):
-                items = {
-                    file: state
-                    for state, files in stage_status.items()
-                    for file in files
-                }
-            if not items:
+            if not stage_status or (
+                isinstance(stage_status, dict) and not any(stage_status.values())
+            ):
                 continue
-            yield stage, items
+            yield stage, stage_status
 
     @classmethod
     def _show_status(cls, status: "DataStatus") -> int:  # noqa: C901
-        git_info = status.pop("git")  # type: ignore[misc]
+        git_info: GitInfo = status.pop("git")  # type: ignore[misc]
         result = dict(cls._process_status(status))
         if not result:
             no_changes = "No changes"
@@ -90,7 +86,16 @@ class CmdDataStatus(CmdBase):
                     ui.write(f"  ({hint})")
 
             if isinstance(stage_status, dict):
-                items = [f"{state}: {file}" for file, state in stage_status.items()]
+                items = [
+                    f"{state}: "
+                    + (
+                        " -> ".join(change.values())
+                        if isinstance(change, dict)
+                        else change
+                    )
+                    for state, changes in stage_status.items()
+                    for change in changes
+                ]
             else:
                 items = stage_status
 
@@ -108,8 +113,10 @@ class CmdDataStatus(CmdBase):
     def run(self) -> int:
         with log_durations(logger.trace, "in data_status"):
             status = self.repo.data_status(
+                targets=self.args.targets,
                 granular=self.args.granular,
                 untracked_files=self.args.untracked_files,
+                remote=self.args.remote,
                 not_in_remote=self.args.not_in_remote,
                 remote_refresh=self.args.remote_refresh,
             )
@@ -126,9 +133,12 @@ class CmdDataStatus(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
+    data_help = "Commands related to data management."
     data_parser = subparsers.add_parser(
         "data",
         parents=[parent_parser],
+        description=append_doc_link(data_help, "data/status"),
+        help=data_help,
         formatter_class=formatter.RawDescriptionHelpFormatter,
     )
     data_subparsers = data_parser.add_subparsers(
@@ -147,6 +157,14 @@ def add_parser(subparsers, parent_parser):
         formatter_class=formatter.RawDescriptionHelpFormatter,
         help=DATA_STATUS_HELP,
     )
+    data_status_parser.add_argument(
+        "targets",
+        nargs="*",
+        help=(
+            "Limit command scope to these tracked files/directories, "
+            ".dvc files and stage names."
+        ),
+    ).complete = completion.FILE  # type: ignore[attr-defined]
     data_status_parser.add_argument(
         "--json",
         action="store_true",
@@ -173,6 +191,12 @@ def add_parser(subparsers, parent_parser):
         nargs="?",
         help="Show untracked files.",
     )
+    data_status_parser.add_argument(
+        "-r",
+        "--remote",
+        help="Remote storage to check (only applicable with --not-in-remote).",
+        metavar="<name>",
+    ).complete = completion.REMOTE
     data_status_parser.add_argument(
         "--not-in-remote",
         action="store_true",

dvc/commands/data_sync.py

@@ -22,17 +22,18 @@ class CmdDataBase(CmdBase):
 
 
 class CmdDataPull(CmdDataBase):
-    def log_summary(self, stats):
+    def log_summary(self, result):
         from dvc.commands.checkout import log_changes
 
-        log_changes(stats)
+        stats = result.pop("stats", {})
+        log_changes(result)
         super().log_summary(stats)
 
     def run(self):
         from dvc.exceptions import CheckoutError, DvcException
 
         try:
-            stats = self.repo.pull(
+            result = self.repo.pull(
                 targets=self.args.targets,
                 jobs=self.args.jobs,
                 remote=self.args.remote,
@@ -46,10 +47,10 @@ class CmdDataPull(CmdDataBase):
                 glob=self.args.glob,
                 allow_missing=self.args.allow_missing,
             )
-            self.log_summary(stats)
+            self.log_summary(result)
         except (CheckoutError, DvcException) as exc:
-            if stats := getattr(exc, "stats", {}):
-                self.log_summary(stats)
+            if result := getattr(exc, "result", {}):
+                self.log_summary(result)
             logger.exception("failed to pull data from the cloud")
             return 1
 
@@ -149,7 +150,7 @@ def add_parser(subparsers, _parent_parser):
     )
     pull_parser.add_argument(
         "-r", "--remote", help="Remote storage to pull from", metavar="<name>"
-    )
+    ).complete = completion.REMOTE
     pull_parser.add_argument(
         "-a",
         "--all-branches",
@@ -194,7 +195,7 @@ def add_parser(subparsers, _parent_parser):
     )
     pull_parser.add_argument(
         "--run-cache",
-        action="store_true",
+        action=argparse.BooleanOptionalAction,
         default=False,
         help="Fetch run history for all stages.",
     )
@@ -224,7 +225,7 @@ def add_parser(subparsers, _parent_parser):
     )
     push_parser.add_argument(
         "-r", "--remote", help="Remote storage to push to", metavar="<name>"
-    )
+    ).complete = completion.REMOTE
     push_parser.add_argument(
         "-a",
         "--all-branches",
@@ -262,7 +263,7 @@ def add_parser(subparsers, _parent_parser):
     )
     push_parser.add_argument(
         "--run-cache",
-        action="store_true",
+        action=argparse.BooleanOptionalAction,
         default=False,
         help="Push run history for all stages.",
     )
@@ -286,7 +287,7 @@ def add_parser(subparsers, _parent_parser):
     )
     fetch_parser.add_argument(
         "-r", "--remote", help="Remote storage to fetch from", metavar="<name>"
-    )
+    ).complete = completion.REMOTE
     fetch_parser.add_argument(
         "-a",
         "--all-branches",
@@ -324,7 +325,7 @@ def add_parser(subparsers, _parent_parser):
     )
     fetch_parser.add_argument(
         "--run-cache",
-        action="store_true",
+        action=argparse.BooleanOptionalAction,
         default=False,
         help="Fetch run history for all stages.",
     )
@@ -379,7 +380,7 @@ def add_parser(subparsers, _parent_parser):
         "--remote",
         help="Remote storage to compare local cache to",
         metavar="<name>",
-    )
+    ).complete = completion.REMOTE
     status_parser.add_argument(
         "-a",
         "--all-branches",
@@ -429,5 +430,11 @@ def add_parser(subparsers, _parent_parser):
         default=False,
         help="Show status in JSON format.",
     )
+    status_parser.add_argument(
+        "--no-updates",
+        dest="check_updates",
+        action="store_false",
+        help="Ignore updates to imported data.",
+    )
 
     status_parser.set_defaults(func=CmdDataStatus)

dvc/commands/dataset.py

@@ -0,0 +1,226 @@
+from typing import TYPE_CHECKING, Optional
+
+from dvc.cli import formatter
+from dvc.cli.command import CmdBase
+from dvc.cli.utils import append_doc_link
+from dvc.exceptions import DvcException
+from dvc.log import logger
+
+if TYPE_CHECKING:
+    from rich.text import Text
+
+    from dvc.repo.datasets import Dataset, FileInfo
+
+logger = logger.getChild(__name__)
+
+
+def diff_files(old: list["FileInfo"], new: list["FileInfo"]) -> dict[str, list[str]]:
+    old_files = {d.relpath: d for d in old}
+    new_files = {d.relpath: d for d in new}
+    rest = old_files.keys() & new_files.keys()
+    return {
+        "added": list(new_files.keys() - old_files.keys()),
+        "deleted": list(old_files.keys() - new_files.keys()),
+        "modified": [p for p in rest if new_files[p] != old_files[p]],
+    }
+
+
+class CmdDatasetAdd(CmdBase):
+    @classmethod
+    def display(cls, name: str, dataset: "Dataset", action: str = "Adding"):
+        from dvc.ui import ui
+
+        assert dataset.lock
+
+        url = dataset.spec.url
+        ver: str = ""
+        if dataset.type == "dc":
+            ver = f"v{dataset.lock.version}"
+        if dataset.type == "dvc":
+            if dataset.lock.path:
+                url = f"{url}:/{dataset.lock.path.lstrip('/')}"
+            if rev := dataset.lock.rev:
+                ver = rev
+
+        ver_part: Optional[Text] = None
+        if ver:
+            ver_part = ui.rich_text.assemble(" @ ", (ver, "repr.number"))
+        text = ui.rich_text.assemble("(", (url, "repr.url"), ver_part or "", ")")
+        ui.write(action, ui.rich_text(name, "cyan"), text, styled=True)
+
+    def run(self):
+        if not self.args.dvc and self.args.rev:
+            raise DvcException("--rev can't be used without --dvc")
+        if not self.args.dvc and self.args.path:
+            raise DvcException("--path can't be used without --dvc")
+
+        d = vars(self.args)
+        for key in ["dvc", "dc", "url"]:
+            if url := d.pop(key, None):
+                d.update({"type": key, "url": url})
+                break
+
+        existing = self.repo.datasets.get(self.args.name)
+        with self.repo.scm_context:
+            if not self.args.force and existing:
+                path = self.repo.fs.relpath(existing.manifest_path)
+                raise DvcException(
+                    f"{self.args.name} already exists in {path}, "
+                    "use the --force to overwrite"
+                )
+            dataset = self.repo.datasets.add(**d)
+            self.display(self.args.name, dataset)
+            return 0
+
+
+class CmdDatasetUpdate(CmdBase):
+    def display(self, name: str, dataset: "Dataset", new: "Dataset"):
+        from dvc.commands.checkout import log_changes
+        from dvc.ui import ui
+
+        action = "Updating"
+        if not dataset.lock:
+            return CmdDatasetAdd.display(name, new, action)
+        if dataset == new:
+            ui.write("[yellow]Nothing to update[/]", styled=True)
+            return None
+
+        assert new.lock
+
+        v: Optional[tuple[str, str]] = None
+        if dataset.type == "dc":
+            assert new.type == "dc"
+            if new.lock.version < dataset.lock.version:
+                action = "Downgrading"
+
+            v = (f"v{dataset.lock.version}", f"v{new.lock.version}")
+        if dataset.type == "dvc":
+            assert new.type == "dvc"
+            v = (f"{dataset.lock.rev_lock[:9]}", f"{new.lock.rev_lock[:9]}")
+
+        if v:
+            part = ui.rich_text.assemble(
+                (v[0], "repr.number"),
+                " -> ",
+                (v[1], "repr.number"),
+            )
+        else:
+            part = ui.rich_text(dataset.spec.url, "repr.url")
+        changes = ui.rich_text.assemble("(", part, ")")
+        ui.write(action, ui.rich_text(name, "cyan"), changes, styled=True)
+        if dataset.type == "url":
+            assert new.type == "url"
+            stats = diff_files(dataset.lock.files, new.lock.files)
+            log_changes(stats)
+        return None
+
+    def run(self):
+        from difflib import get_close_matches
+
+        from dvc.repo.datasets import DatasetNotFoundError
+        from dvc.ui import ui
+
+        version = None
+        if self.args.rev:
+            try:
+                version = int(self.args.rev.lstrip("v"))
+            except ValueError:
+                version = self.args.rev
+
+        d = vars(self.args) | {"version": version}
+        with self.repo.scm_context:
+            try:
+                dataset, new = self.repo.datasets.update(**d)
+            except DatasetNotFoundError:
+                logger.exception("")
+                if matches := get_close_matches(self.args.name, self.repo.datasets):
+                    ui.write(
+                        "did you mean?",
+                        ui.rich_text(matches[0], "cyan"),
+                        stderr=True,
+                        styled=True,
+                    )
+                return 1
+            self.display(self.args.name, dataset, new)
+            return 0
+
+
+def add_parser(subparsers, parent_parser):
+    ds_parser = subparsers.add_parser(
+        "dataset",
+        aliases=["ds"],
+        parents=[parent_parser],
+        formatter_class=formatter.RawDescriptionHelpFormatter,
+    )
+    ds_subparsers = ds_parser.add_subparsers(
+        dest="cmd",
+        help="Use `dvc dataset CMD --help` to display command-specific help.",
+        required=True,
+    )
+
+    dataset_add_help = "Add a dataset."
+    ds_add_parser = ds_subparsers.add_parser(
+        "add",
+        parents=[parent_parser],
+        description=append_doc_link(dataset_add_help, "dataset/add"),
+        formatter_class=formatter.RawTextHelpFormatter,
+        help=dataset_add_help,
+    )
+
+    url_exclusive_group = ds_add_parser.add_mutually_exclusive_group(required=True)
+    url_exclusive_group.add_argument(
+        "--dc", metavar="name", help="Name of the DataChain dataset to track"
+    )
+    url_exclusive_group.add_argument(
+        "--dvc",
+        help="Path or URL to a Git/DVC repository to track",
+        metavar="url",
+    )
+    url_exclusive_group.add_argument(
+        "--url",
+        help="""\
+URL of a cloud-versioned remote to track. Supported URLs:
+
+s3://bucket/key/path
+gs://bucket/path/to/file/or/dir
+azure://mycontainer/path
+remote://remote_name/path/to/file/or/dir (see `dvc remote`)
+""",
+    )
+    ds_add_parser.add_argument("name", help="Name of the dataset to add")
+    ds_add_parser.add_argument(
+        "--rev",
+        help="Git revision, e.g. SHA, branch, tag (only applicable with --dvc)",
+        metavar="<commit>",
+    )
+    ds_add_parser.add_argument(
+        "--path",
+        help="Path to a file or a directory within a git repository "
+        "(only applicable with --dvc)",
+    )
+    ds_add_parser.add_argument(
+        "-f",
+        "--force",
+        action="store_true",
+        default=False,
+        help="Overwrite existing dataset",
+    )
+    ds_add_parser.set_defaults(func=CmdDatasetAdd)
+
+    dataset_update_help = "Update a dataset."
+    ds_update_parser = ds_subparsers.add_parser(
+        "update",
+        parents=[parent_parser],
+        description=append_doc_link(dataset_update_help, "dataset/update"),
+        formatter_class=formatter.RawDescriptionHelpFormatter,
+        help=dataset_update_help,
+    )
+    ds_update_parser.add_argument("name", help="Name of the dataset to update")
+    ds_update_parser.add_argument(
+        "--rev",
+        "--version",
+        nargs="?",
+        help="DataChain dataset version or Git revision (e.g. SHA, branch, tag)",
+        metavar="<version>",
+    )
+    ds_update_parser.set_defaults(func=CmdDatasetUpdate)

dvc/commands/du.py

@@ -60,7 +60,7 @@ def add_parser(subparsers, parent_parser):
         "--remote",
         type=str,
         help="Remote name to set as a default in the target repository.",
-    )
+    ).complete = completion.REMOTE
     du_parser.add_argument(
         "--remote-config",
         type=str,

dvc/commands/experiments/__init__.py

@@ -58,6 +58,15 @@ def add_parser(subparsers, parent_parser):
     hide_subparsers_from_help(experiments_subparsers)
 
 
+def add_keep_selection_flag(experiments_subcmd_parser):
+    experiments_subcmd_parser.add_argument(
+        "--keep",
+        action="store_true",
+        default=False,
+        help="Keep the selected experiments instead of removing them.",
+    )
+
+
 def add_rev_selection_flags(
     experiments_subcmd_parser, command: str, default: bool = True
 ):

dvc/commands/experiments/pull.py

@@ -1,4 +1,6 @@
-from dvc.cli import formatter
+import argparse
+
+from dvc.cli import completion, formatter
 from dvc.cli.command import CmdBase
 from dvc.cli.utils import append_doc_link
 from dvc.log import logger
@@ -69,7 +71,7 @@ def add_parser(experiments_subparsers, parent_parser):
         dest="dvc_remote",
         metavar="<name>",
         help="Name of the DVC remote to use when pulling cached outputs.",
-    )
+    ).complete = completion.REMOTE
     experiments_pull_parser.add_argument(
         "-j",
         "--jobs",
@@ -81,7 +83,7 @@ def add_parser(experiments_subparsers, parent_parser):
     )
     experiments_pull_parser.add_argument(
         "--run-cache",
-        action="store_true",
+        action=argparse.BooleanOptionalAction,
         default=False,
         help="Pull run history for all stages.",
     )

dvc/commands/experiments/push.py

@@ -1,4 +1,5 @@
-from typing import Any, Dict
+import argparse
+from typing import Any
 
 from dvc.cli import completion, formatter
 from dvc.cli.command import CmdBase
@@ -11,7 +12,7 @@ logger = logger.getChild(__name__)
 
 class CmdExperimentsPush(CmdBase):
     @staticmethod
-    def log_result(result: Dict[str, Any], remote: str):
+    def log_result(result: dict[str, Any], remote: str):
         from dvc.utils import humanize
 
         def join_exps(exps):
@@ -110,7 +111,7 @@ def add_parser(experiments_subparsers, parent_parser):
         dest="dvc_remote",
         metavar="<name>",
         help="Name of the DVC remote to use when pushing cached outputs.",
-    )
+    ).complete = completion.REMOTE
     experiments_push_parser.add_argument(
         "-j",
         "--jobs",
@@ -120,7 +121,7 @@ def add_parser(experiments_subparsers, parent_parser):
     )
     experiments_push_parser.add_argument(
         "--run-cache",
-        action="store_true",
+        action=argparse.BooleanOptionalAction,
         default=False,
         help="Push run history for all stages.",
     )

dvc/commands/experiments/remove.py

@@ -34,6 +34,7 @@ class CmdExperimentsRemove(CmdBase):
             num=self.args.num,
             queue=self.args.queue,
             git_remote=self.args.git_remote,
+            keep=self.args.keep,
         )
         if removed:
             ui.write(f"Removed experiments: {humanize.join(map(repr, removed))}")
@@ -44,7 +45,7 @@ class CmdExperimentsRemove(CmdBase):
 
 
 def add_parser(experiments_subparsers, parent_parser):
-    from . import add_rev_selection_flags
+    from . import add_keep_selection_flag, add_rev_selection_flags
 
     EXPERIMENTS_REMOVE_HELP = "Remove experiments."
     experiments_remove_parser = experiments_subparsers.add_parser(
@@ -57,6 +58,7 @@ def add_parser(experiments_subparsers, parent_parser):
     )
     remove_group = experiments_remove_parser.add_mutually_exclusive_group()
     add_rev_selection_flags(experiments_remove_parser, "Remove", False)
+    add_keep_selection_flag(experiments_remove_parser)
     remove_group.add_argument(
         "--queue", action="store_true", help="Remove all queued experiments."
     )

dvc/commands/experiments/run.py

@@ -20,6 +20,7 @@ class CmdExperimentsRun(CmdRepro):
             tmp_dir=self.args.tmp_dir,
             copy_paths=self.args.copy_paths,
             message=self.args.message,
+            no_hydra=self.args.no_hydra,
             **self._common_kwargs,
         )
 
@@ -108,7 +109,13 @@ def _add_run_common(parser):
         help="Custom commit message to use when committing the experiment.",
     )
     parser.add_argument(
-        "-M",  # obsolete
-        dest="message",
-        help=argparse.SUPPRESS,
+        "--no-hydra",
+        action="store_true",
+        default=False,
+        help=(
+            "Disables automatically updating `params.yaml` with Hydra configuration. "
+            " You can still use `--set-param` to update individual params if needed."
+            " Default is False."
+        ),
     )
+    parser.add_argument("-M", dest="message", help=argparse.SUPPRESS)  # obsolete

dvc/commands/experiments/save.py

@@ -93,9 +93,5 @@ def add_parser(experiments_subparsers, parent_parser):
         default=None,
         help="Custom commit message to use when committing the experiment.",
     )
-    save_parser.add_argument(
-        "-M",  # obsolete
-        dest="message",
-        help=argparse.SUPPRESS,
-    )
+    save_parser.add_argument("-M", dest="message", help=argparse.SUPPRESS)  # obsolete
     save_parser.set_defaults(func=CmdExperimentsSave)

dvc/commands/experiments/show.py

@@ -1,7 +1,8 @@
 import argparse
 import re
+from collections.abc import Collection, Iterable
 from datetime import date, datetime
-from typing import TYPE_CHECKING, Dict, Iterable
+from typing import TYPE_CHECKING
 
 from funcy import lmap
 
@@ -57,18 +58,17 @@ def baseline_styler(typ):
 
 def show_experiments(
     td: "TabularData",
-    headers: Dict[str, Iterable[str]],
-    keep=None,
-    drop=None,
+    headers: dict[str, Iterable[str]],
+    keep: Collection[str] = (),
+    drop: Collection[str] = (),
     pager=True,
     csv=False,
     markdown=False,
     **kwargs,
 ):
     if keep:
-        for col in td.keys():  # noqa: SIM118
-            if re.match(keep, col):
-                td.protect(col)
+        keep_re = re.compile("|".join(keep))
+        td.protect(*(col for col in td.keys() if keep_re.match(col)))  # noqa: SIM118
 
     for col in ("State", "Executor"):
         if td.is_empty(col):
@@ -109,8 +109,9 @@ def show_experiments(
         td.drop_duplicates("cols", ignore_empty=False)
 
     cols_to_drop = set()
-    if drop is not None:
-        cols_to_drop = {col for col in td.keys() if re.match(drop, col)}  # noqa: SIM118
+    if drop:
+        drop_re = re.compile("|".join(drop))
+        cols_to_drop = {col for col in td.keys() if drop_re.match(col)}  # noqa: SIM118
     td.drop(*cols_to_drop)
 
     td.render(
@@ -149,6 +150,7 @@ class CmdExperimentsShow(CmdBase):
                 all_commits=self.args.all_commits,
                 hide_queued=self.args.hide_queued,
                 hide_failed=self.args.hide_failed,
+                hide_workspace=self.args.hide_workspace,
                 revs=self.args.rev,
                 num=self.args.num,
                 sha_only=self.args.sha,
@@ -235,11 +237,15 @@ def add_parser(experiments_subparsers, parent_parser):
     )
     experiments_show_parser.add_argument(
         "--drop",
+        action="append",
+        default=[],
         help="Remove the columns matching the specified regex pattern.",
         metavar="<regex_pattern>",
     )
     experiments_show_parser.add_argument(
         "--keep",
+        action="append",
+        default=[],
         help="Preserve the columns matching the specified regex pattern.",
         metavar="<regex_pattern>",
     )
@@ -278,6 +284,12 @@ def add_parser(experiments_subparsers, parent_parser):
         default=False,
         help="Hide queued experiments in the table.",
     )
+    experiments_show_parser.add_argument(
+        "--hide-workspace",
+        action="store_true",
+        default=False,
+        help="Hide workspace row in the table.",
+    )
     experiments_show_parser.add_argument(
         "--json",
         action="store_true",

dvc/commands/gc.py

@@ -1,6 +1,6 @@
 import os
 
-from dvc.cli import formatter
+from dvc.cli import completion, formatter
 from dvc.cli.command import CmdBase
 from dvc.cli.utils import append_doc_link
 from dvc.log import logger
@@ -40,9 +40,16 @@ class CmdGC(CmdBase):
             msg += " and all git commits"
         else:
             if self.args.all_branches and self.args.all_tags:
-                msg += " and all git branches and tags"
+                if self.args.num:
+                    msg += f" and last {self.args.num} commits from all git branches"
+                    msg += " and all git tags"
+                else:
+                    msg += " and all git branches and tags"
             elif self.args.all_branches:
-                msg += " and all git branches"
+                if self.args.num:
+                    msg += f" and last {self.args.num} commits from all git branches"
+                else:
+                    msg += " and all git branches"
             elif self.args.all_tags:
                 msg += " and all git tags"
             if self.args.commit_date:
@@ -54,13 +61,13 @@ class CmdGC(CmdBase):
             msg += " and all experiments"
 
         if self.args.not_in_remote:
-            msg += " that are present in the DVC remote"
+            msg += " that are not present in the DVC remote"
 
         if self.args.repos:
             msg += " of the current and the following repos:"
 
             for repo_path in self.args.repos:
-                msg += "\n  - %s" % os.path.abspath(repo_path)
+                msg += f"\n  - {os.path.abspath(repo_path)}"
         else:
             msg += " of the current repo."
 
@@ -126,8 +133,8 @@ def add_parser(subparsers, parent_parser):
         metavar="<num>",
         help=(
             "Keep data files used in the last `num` commits "
-            "starting from the `--rev` <commit>. "
-            "Only used if `--rev` is also provided. "
+            "starting from each selected revision root. "
+            "Can be used with `--rev` and `--all-branches`. "
             "Defaults to `1`."
         ),
     )
@@ -188,7 +195,7 @@ def add_parser(subparsers, parent_parser):
         "--remote",
         help="Remote storage to collect garbage in",
         metavar="<name>",
-    )
+    ).complete = completion.REMOTE
     gc_parser.add_argument(
         "--skip-failed",
         action="store_true",

dvc/commands/get.py

@@ -12,7 +12,13 @@ class CmdGet(CmdBaseNoRepo):
         from dvc.api import get_url
         from dvc.ui import ui
 
-        url = get_url(self.args.path, repo=self.args.url, rev=self.args.rev)
+        url = get_url(
+            self.args.path,
+            repo=self.args.url,
+            rev=self.args.rev,
+            remote=self.args.remote,
+            remote_config=self.args.remote_config,
+        )
         ui.write(url, force=True)
 
         return 0
@@ -115,7 +121,7 @@ def add_parser(subparsers, parent_parser):
         "--remote",
         type=str,
         help="Remote name to set as a default in the target repository.",
-    )
+    ).complete = completion.REMOTE
     get_parser.add_argument(
         "--remote-config",
         type=str,

dvc/commands/imp.py

@@ -115,7 +115,7 @@ def add_parser(subparsers, parent_parser):
         "--remote",
         type=str,
         help="Remote name to set as a default in the target repository.",
-    )
+    ).complete = completion.REMOTE
     import_parser.add_argument(
         "--remote-config",
         type=str,

dvc/commands/imp_url.py

@@ -73,7 +73,7 @@ def add_parser(subparsers, parent_parser):
         "--remote",
         help="Remote storage to download to",
         metavar="<name>",
-    )
+    ).complete = completion.REMOTE
     no_download_exec_group = import_parser.add_mutually_exclusive_group()
     no_download_exec_group.add_argument(
         "--no-exec",

dvc/commands/init.py

@@ -29,7 +29,7 @@ def _welcome_message():
         "{yellow}------------{nc}\n"
         f"- Check out the documentation: {fmt_link('https://dvc.org/doc')}\n"
         f"- Get help and share ideas: {fmt_link('https://dvc.org/chat')}\n"
-        f"- Star us on GitHub: {fmt_link('https://github.com/iterative/dvc')}"
+        f"- Star us on GitHub: {fmt_link('https://github.com/treeverse/dvc')}"
     ).format(yellow=colorama.Fore.YELLOW, nc=colorama.Fore.RESET)
 
     ui.write(msg)
@@ -42,7 +42,7 @@ class CmdInit(CmdBaseNoRepo):
 
         try:
             with Repo.init(
-                ".",
+                self.args.directory,
                 no_scm=self.args.no_scm,
                 force=self.args.force,
                 subdir=self.args.subdir,
@@ -57,10 +57,11 @@ class CmdInit(CmdBaseNoRepo):
 
 def add_parser(subparsers, parent_parser):
     """Setup parser for `dvc init`."""
-    INIT_HELP = "Initialize DVC in the current directory."
+    INIT_HELP = "Initialize DVC repository."
     INIT_DESCRIPTION = (
-        "Initialize DVC in the current directory. Expects directory\n"
-        "to be a Git repository unless --no-scm option is specified."
+        "Initialize DVC repository in the given directory (defaults to the current "
+        "working directory).\n"
+        "Expects directory to be a Git repository unless --no-scm option is specified."
     )
 
     init_parser = subparsers.add_parser(
@@ -70,6 +71,14 @@ def add_parser(subparsers, parent_parser):
         help=INIT_HELP,
         formatter_class=formatter.RawDescriptionHelpFormatter,
     )
+    init_parser.add_argument(
+        "directory",
+        nargs="?",
+        default=".",
+        help=(
+            "Directory to initialize DVC in. Defaults to the current working directory."
+        ),
+    )
     init_parser.add_argument(
         "--no-scm",
         action="store_true",

dvc/commands/ls/__init__.py

@@ -1,3 +1,5 @@
+from typing import Callable
+
 from dvc.cli import completion, formatter
 from dvc.cli.command import CmdBaseNoRepo
 from dvc.cli.utils import DictAction, append_doc_link
@@ -9,54 +11,173 @@ from dvc.ui import ui
 logger = logger.getChild(__name__)
 
 
-def _format_entry(entry, fmt):
-    from dvc.utils.humanize import naturalsize
+def _get_formatter(with_color: bool = False) -> Callable[[dict], str]:
+    def fmt(entry: dict) -> str:
+        return entry["path"]
 
-    size = entry.get("size")
-    if size is None:
-        size = ""
-    else:
-        size = naturalsize(size)
-    return size, fmt(entry)
-
-
-def show_entries(entries, with_color=False, with_size=False):
     if with_color:
         ls_colors = LsColors()
-        fmt = ls_colors.format
-    else:
+        return ls_colors.format
 
-        def fmt(entry):
-            return entry["path"]
+    return fmt
 
+
+def _format_entry(entry, name, with_size=True, with_hash=False):
+    from dvc.utils.humanize import naturalsize
+
+    ret = []
     if with_size:
-        ui.table([_format_entry(entry, fmt) for entry in entries])
+        size = entry.get("size")
+        if size is None or (size <= 0 and entry.get("isdir")):
+            size = ""
+        else:
+            size = naturalsize(size)
+        ret.append(size)
+    if with_hash:
+        md5 = entry.get("md5", "")
+        ret.append(md5)
+    ret.append(name)
+    return ret
+
+
+def show_entries(entries, with_color=False, with_size=False, with_hash=False):
+    fmt = _get_formatter(with_color)
+    if with_size or with_hash:
+        colalign = ("right",) if with_size else None
+        ui.table(
+            [
+                _format_entry(
+                    entry,
+                    fmt(entry),
+                    with_size=with_size,
+                    with_hash=with_hash,
+                )
+                for entry in entries
+            ],
+            colalign=colalign,
+        )
         return
 
     # NOTE: this is faster than ui.table for very large number of entries
     ui.write("\n".join(fmt(entry) for entry in entries))
 
 
+class TreePart:
+    Edge = "├── "
+    Line = "│   "
+    Corner = "└── "
+    Blank = "    "
+
+
+def _build_tree_structure(
+    entries, with_color=False, with_size=False, with_hash=False, _depth=0, _prefix=""
+):
+    rows = []
+    fmt = _get_formatter(with_color)
+
+    num_entries = len(entries)
+    for i, (name, entry) in enumerate(entries.items()):
+        entry["path"] = name
+        is_last = i >= num_entries - 1
+        tree_part = ""
+        if _depth > 0:
+            tree_part = TreePart.Corner if is_last else TreePart.Edge
+
+        row = _format_entry(
+            entry,
+            _prefix + tree_part + fmt(entry),
+            with_size=with_size,
+            with_hash=with_hash,
+        )
+        rows.append(row)
+
+        if contents := entry.get("contents"):
+            new_prefix = _prefix
+            if _depth > 0:
+                new_prefix += TreePart.Blank if is_last else TreePart.Line
+            new_rows = _build_tree_structure(
+                contents,
+                with_color=with_color,
+                with_size=with_size,
+                with_hash=with_hash,
+                _depth=_depth + 1,
+                _prefix=new_prefix,
+            )
+            rows.extend(new_rows)
+
+    return rows
+
+
+def show_tree(entries, with_color=False, with_size=False, with_hash=False):
+    rows = _build_tree_structure(
+        entries,
+        with_color=with_color,
+        with_size=with_size,
+        with_hash=with_hash,
+    )
+
+    colalign = ("right",) if with_size else None
+    ui.table(rows, colalign=colalign, preserve_whitespace=True)
+
+
 class CmdList(CmdBaseNoRepo):
-    def run(self):
+    def _show_tree(self):
+        from dvc.repo.ls import ls_tree
+
+        entries = ls_tree(
+            self.args.url,
+            self.args.path,
+            rev=self.args.rev,
+            dvc_only=self.args.dvc_only,
+            config=self.args.config,
+            remote=self.args.remote,
+            remote_config=self.args.remote_config,
+            maxdepth=self.args.level,
+        )
+        show_tree(
+            entries,
+            with_color=True,
+            with_size=self.args.size,
+            with_hash=self.args.show_hash,
+        )
+        return 0
+
+    def _show_list(self):
         from dvc.repo import Repo
 
-        try:
-            entries = Repo.ls(
-                self.args.url,
-                self.args.path,
-                rev=self.args.rev,
-                recursive=self.args.recursive,
-                dvc_only=self.args.dvc_only,
-                config=self.args.config,
-                remote=self.args.remote,
-                remote_config=self.args.remote_config,
+        entries = Repo.ls(
+            self.args.url,
+            self.args.path,
+            rev=self.args.rev,
+            recursive=self.args.recursive,
+            dvc_only=self.args.dvc_only,
+            config=self.args.config,
+            remote=self.args.remote,
+            remote_config=self.args.remote_config,
+            maxdepth=self.args.level,
+        )
+        if self.args.json:
+            ui.write_json(entries)
+        elif entries:
+            show_entries(
+                entries,
+                with_color=True,
+                with_size=self.args.size,
+                with_hash=self.args.show_hash,
             )
-            if self.args.json:
-                ui.write_json(entries)
-            elif entries:
-                show_entries(entries, with_color=True, with_size=self.args.size)
-            return 0
+        return 0
+
+    def run(self):
+        if self.args.tree and self.args.json:
+            raise DvcException("Cannot use --tree and --json options together.")
+
+        try:
+            if self.args.tree:
+                return self._show_tree()
+            return self._show_list()
+        except FileNotFoundError:
+            logger.exception("")
+            return 1
         except DvcException:
             logger.exception("failed to list '%s'", self.args.url)
             return 1
@@ -82,6 +203,19 @@ def add_parser(subparsers, parent_parser):
         action="store_true",
         help="Recursively list files.",
     )
+    list_parser.add_argument(
+        "-T",
+        "--tree",
+        action="store_true",
+        help="Recurse into directories as a tree.",
+    )
+    list_parser.add_argument(
+        "-L",
+        "--level",
+        metavar="depth",
+        type=int,
+        help="Limit the depth of recursion.",
+    )
     list_parser.add_argument(
         "--dvc-only", action="store_true", help="Show only DVC outputs."
     )
@@ -108,7 +242,7 @@ def add_parser(subparsers, parent_parser):
         "--remote",
         type=str,
         help="Remote name to set as a default in the target repository.",
-    )
+    ).complete = completion.REMOTE
     list_parser.add_argument(
         "--remote-config",
         type=str,
@@ -119,10 +253,12 @@ def add_parser(subparsers, parent_parser):
             "specified by '--remote') in the target repository."
         ),
     )
+    list_parser.add_argument("--size", action="store_true", help="Show sizes.")
     list_parser.add_argument(
-        "--size",
+        "--show-hash",
+        help="Display hash value for each item.",
         action="store_true",
-        help="Show sizes.",
+        default=False,
     )
     list_parser.add_argument(
         "path",

dvc/commands/ls/ls_colors.py

@@ -20,7 +20,7 @@ class LsColors:
             else:
                 self._codes[code] = color
 
-    def format(self, entry):  # noqa: A003
+    def format(self, entry):
         text = entry["path"]
 
         if entry.get("isout", False) and "out" in self._codes:
@@ -32,7 +32,9 @@ class LsColors:
         if entry.get("isexec", False):
             return self._format(text, code="ex")
 
-        _, ext = os.path.splitext(text)
+        stem, ext = os.path.splitext(text)
+        if not ext and stem.startswith("."):
+            ext = stem
         return self._format(text, ext=ext)
 
     def _format(self, text, code=None, ext=None):

dvc/commands/ls_url.py

@@ -3,26 +3,45 @@ from dvc.cli.command import CmdBaseNoRepo
 from dvc.cli.utils import DictAction, append_doc_link
 from dvc.log import logger
 
-from .ls import show_entries
+from .ls import show_entries, show_tree
 
 logger = logger.getChild(__name__)
 
 
 class CmdListUrl(CmdBaseNoRepo):
-    def run(self):
-        from dvc.config import Config
+    def _show_tree(self, config):
+        from dvc.fs import parse_external_url
+        from dvc.repo.ls import _ls_tree
+
+        fs, fs_path = parse_external_url(
+            self.args.url, fs_config=self.args.fs_config, config=config
+        )
+        entries = _ls_tree(fs, fs_path, maxdepth=self.args.level)
+        show_tree(entries, with_color=True, with_size=self.args.size)
+        return 0
+
+    def _show_list(self, config):
         from dvc.repo import Repo
 
         entries = Repo.ls_url(
             self.args.url,
             recursive=self.args.recursive,
+            maxdepth=self.args.level,
             fs_config=self.args.fs_config,
-            config=Config.from_cwd(),
+            config=config,
         )
         if entries:
             show_entries(entries, with_color=True, with_size=self.args.size)
         return 0
 
+    def run(self):
+        from dvc.config import Config
+
+        config = Config.from_cwd()
+        if self.args.tree:
+            return self._show_tree(config=config)
+        return self._show_list(config=config)
+
 
 def add_parser(subparsers, parent_parser):
     LS_HELP = "List directory contents from URL."
@@ -38,16 +57,22 @@ def add_parser(subparsers, parent_parser):
         "url", help="See `dvc import-url -h` for full list of supported URLs."
     )
     lsurl_parser.add_argument(
-        "-R",
-        "--recursive",
-        action="store_true",
-        help="Recursively list files.",
+        "-R", "--recursive", action="store_true", help="Recursively list files."
     )
     lsurl_parser.add_argument(
-        "--size",
+        "-T",
+        "--tree",
         action="store_true",
-        help="Show sizes.",
+        help="Recurse into directories as a tree.",
     )
+    lsurl_parser.add_argument(
+        "-L",
+        "--level",
+        metavar="depth",
+        type=int,
+        help="Limit the depth of recursion.",
+    )
+    lsurl_parser.add_argument("--size", action="store_true", help="Show sizes.")
     lsurl_parser.add_argument(
         "--fs-config",
         type=str,

dvc/commands/plots.py

@@ -1,6 +1,6 @@
 import argparse
 import os
-from typing import TYPE_CHECKING, Dict, List, Optional
+from typing import TYPE_CHECKING, Optional
 
 from funcy import compact, first, get_in
 
@@ -20,35 +20,26 @@ logger = logger.getChild(__name__)
 
 
 def _show_json(
-    renderers_with_errors: List["RendererWithErrors"],
+    renderers_with_errors: list["RendererWithErrors"],
     split=False,
-    errors: Optional[Dict[str, Exception]] = None,
+    errors: Optional[dict[str, Exception]] = None,
 ):
     from dvc.render.convert import to_json
     from dvc.utils.serialize import encode_exception
 
-    all_errors: List[Dict] = []
+    all_errors: list[dict] = []
     data = {}
 
     for renderer, src_errors, def_errors in renderers_with_errors:
         name = renderer.name
         data[name] = to_json(renderer, split)
         all_errors.extend(
-            {
-                "name": name,
-                "rev": rev,
-                "source": source,
-                **encode_exception(e),
-            }
+            {"name": name, "rev": rev, "source": source, **encode_exception(e)}
             for rev, per_rev_src_errors in src_errors.items()
             for source, e in per_rev_src_errors.items()
         )
         all_errors.extend(
-            {
-                "name": name,
-                "rev": rev,
-                **encode_exception(e),
-            }
+            {"name": name, "rev": rev, **encode_exception(e)}
             for rev, e in def_errors.items()
         )
 
@@ -81,7 +72,7 @@ class CmdPlots(CmdBase):
                 html_template_path = os.path.join(self.repo.dvc_dir, html_template_path)
         return html_template_path
 
-    def run(self) -> int:  # noqa: C901, PLR0911, PLR0912
+    def run(self) -> int:  # noqa: C901, PLR0911
         from pathlib import Path
 
         from dvc.render.match import match_defs_renderers
@@ -101,10 +92,7 @@ class CmdPlots(CmdBase):
                 return 1
 
         try:
-            plots_data = self._func(
-                targets=self.args.targets,
-                props=self._props(),
-            )
+            plots_data = self._func(targets=self.args.targets, props=self._props())
 
             if not plots_data and not self.args.json:
                 ui.error_write(

dvc/commands/queue/__init__.py

@@ -2,14 +2,7 @@ from dvc.cli import formatter
 from dvc.cli.utils import append_doc_link
 from dvc.commands.queue import kill, logs, remove, start, status, stop
 
-SUB_COMMANDS = [
-    start,
-    stop,
-    status,
-    logs,
-    remove,
-    kill,
-]
+SUB_COMMANDS = [start, stop, status, logs, remove, kill]
 
 
 def add_parser(subparsers, parent_parser):

dvc/commands/queue/logs.py

@@ -45,9 +45,5 @@ def add_parser(queue_subparsers, parent_parser):
         ),
         action="store_true",
     )
-    queue_logs_parser.add_argument(
-        "task",
-        help="Task to show.",
-        metavar="<task>",
-    )
+    queue_logs_parser.add_argument("task", help="Task to show.", metavar="<task>")
     queue_logs_parser.set_defaults(func=CmdQueueLogs)

dvc/commands/remote.py

@@ -1,4 +1,4 @@
-from dvc.cli import formatter
+from dvc.cli import completion, formatter
 from dvc.cli.utils import append_doc_link
 from dvc.commands.config import CmdConfig
 from dvc.ui import ui
@@ -110,8 +110,16 @@ class CmdRemoteDefault(CmdRemote):
 class CmdRemoteList(CmdRemote):
     def run(self):
         conf = self.config.read(self.args.level)
+        default_remote = conf["core"].get("remote")
+
         for name, remote_conf in conf["remote"].items():
-            ui.write(name, remote_conf["url"], sep="\t")
+            if name == default_remote:
+                text = f"{name}\t{remote_conf['url']}\t(default)"
+                color = "green"
+            else:
+                text = f"{name}\t{remote_conf['url']}"
+                color = ""
+            ui.write(ui.rich_text(text, style=color), styled=True)
         return 0
 
 
@@ -202,7 +210,9 @@ def add_parser(subparsers, parent_parser):
         help=REMOTE_DEFAULT_HELP,
         formatter_class=formatter.RawDescriptionHelpFormatter,
     )
-    remote_default_parser.add_argument("name", nargs="?", help="Name of the remote")
+    remote_default_parser.add_argument(
+        "name", nargs="?", help="Name of the remote"
+    ).complete = completion.REMOTE
     remote_default_parser.add_argument(
         "-u",
         "--unset",
@@ -220,7 +230,9 @@ def add_parser(subparsers, parent_parser):
         help=REMOTE_MODIFY_HELP,
         formatter_class=formatter.RawDescriptionHelpFormatter,
     )
-    remote_modify_parser.add_argument("name", help="Name of the remote")
+    remote_modify_parser.add_argument(
+        "name", help="Name of the remote"
+    ).complete = completion.REMOTE
     remote_modify_parser.add_argument("option", help="Name of the option to modify.")
     remote_modify_parser.add_argument(
         "value", nargs="?", help="(optional) Value of the option."
@@ -252,7 +264,9 @@ def add_parser(subparsers, parent_parser):
         help=REMOTE_REMOVE_HELP,
         formatter_class=formatter.RawDescriptionHelpFormatter,
     )
-    remote_remove_parser.add_argument("name", help="Name of the remote to remove.")
+    remote_remove_parser.add_argument(
+        "name", help="Name of the remote to remove."
+    ).complete = completion.REMOTE
     remote_remove_parser.set_defaults(func=CmdRemoteRemove)
     REMOTE_RENAME_HELP = "Rename a DVC remote"
     remote_rename_parser = remote_subparsers.add_parser(

dvc/commands/repro.py

@@ -117,10 +117,7 @@ and then the stage name name.
         "--pull",
         action="store_true",
         default=False,
-        help=(
-            "Try automatically pulling missing cache for outputs restored "
-            "from the run-cache."
-        ),
+        help="Try automatically pulling missing data.",
     )
     repro_parser.add_argument(
         "--allow-missing",

dvc/commands/stage.py

@@ -1,8 +1,9 @@
 import argparse
 import logging
+from collections.abc import Iterable
 from contextlib import contextmanager
 from itertools import chain, filterfalse
-from typing import TYPE_CHECKING, Dict, Iterable, List
+from typing import TYPE_CHECKING
 
 from dvc.cli import completion, formatter
 from dvc.cli.command import CmdBase
@@ -33,7 +34,7 @@ def generate_description(stage: "Stage") -> str:
     def is_plot_or_metric(out: "Output"):
         return bool(out.plot) or bool(out.metric)
 
-    desc: List[str] = []
+    desc: list[str] = []
 
     outs = list(filterfalse(is_plot_or_metric, stage.outs))
     if outs:
@@ -55,7 +56,7 @@ def prepare_stages_data(
     stages: Iterable["Stage"],
     description: bool = True,
     max_length: int = MAX_TEXT_LENGTH,
-) -> Dict[str, str]:
+) -> dict[str, str]:
     return {
         stage.addressing: (
             prepare_description(stage, max_length=max_length) if description else ""
@@ -67,7 +68,7 @@ def prepare_stages_data(
 class CmdStageList(CmdBase):
     def _get_stages(self) -> Iterable["Stage"]:
         if self.args.all:
-            stages: List["Stage"] = self.repo.index.stages
+            stages: list[Stage] = self.repo.index.stages
             logger.trace("%d no. of stages found", len(stages))
             return stages
 
@@ -96,7 +97,7 @@ class CmdStageList(CmdBase):
         return 0
 
 
-def parse_cmd(commands: List[str]) -> str:
+def parse_cmd(commands: list[str]) -> str:
     """
     We need to take into account two cases:
 

dvc/commands/status.py

@@ -60,18 +60,19 @@ class CmdDataStatus(CmdDataBase):
                     all_commits=self.args.all_commits,
                     with_deps=self.args.with_deps,
                     recursive=self.args.recursive,
+                    check_updates=self.args.check_updates,
                 )
             except DvcException:
                 logger.exception("")
                 return 1
 
-            if self.args.quiet:
-                return bool(st)
-
             if self.args.json:
                 ui.write_json(st)
                 return 0
 
+            if self.args.quiet:
+                return int(bool(st))
+
             if st:
                 self._show(st, indent)
                 return 0

dvc/commands/studio.py

@@ -5,6 +5,7 @@ from funcy import get_in
 from dvc.cli import formatter
 from dvc.cli.utils import append_doc_link
 from dvc.commands.config import CmdConfig
+from dvc.exceptions import DvcException
 from dvc.log import logger
 
 logger = logger.getChild(__name__)
@@ -12,22 +13,35 @@ logger = logger.getChild(__name__)
 
 class CmdStudioLogin(CmdConfig):
     def run(self):
-        from dvc_studio_client.auth import StudioAuthError, get_access_token
-
         from dvc.env import DVC_STUDIO_URL
         from dvc.ui import ui
         from dvc.utils.studio import STUDIO_URL
+        from dvc_studio_client.auth import StudioAuthError, get_access_token
 
+        studio = self.config.get("studio", {})
         name = self.args.name
-        hostname = self.args.hostname or os.environ.get(DVC_STUDIO_URL) or STUDIO_URL
+        hostname = (
+            self.args.hostname
+            or os.environ.get(DVC_STUDIO_URL)
+            or studio.get("url")
+            or STUDIO_URL
+        )
         scopes = self.args.scopes
 
+        if studio.get("url", hostname) == hostname and "token" in studio:
+            raise DvcException(
+                "Token already exists. "
+                "To login with a different token, "
+                "logout using 'dvc studio logout'."
+            )
+
+        open_browser = not self.args.no_open
         try:
-            token_name, access_token = get_access_token(
+            _, access_token = get_access_token(
                 token_name=name,
                 hostname=hostname,
                 scopes=scopes,
-                use_device_code=self.args.no_open,
+                open_browser=open_browser,
                 client_name="DVC",
             )
         except StudioAuthError as e:
@@ -35,17 +49,26 @@ class CmdStudioLogin(CmdConfig):
             return 1
 
         self.save_config(hostname, access_token)
-        ui.write(
-            "Authentication has been successfully completed."
-            "The generated token will now be accessible as"
-            f" {token_name} in the user's Studio profile."
-        )
+
+        if not self.config["exp"].get("auto_push", True):
+            from dvc.ui import ui
+
+            ui.warn(
+                "exp.auto_push is disabled. \n"
+                "Enable with 'dvc config exp.auto_push true' "
+                "to automatically push experiments to Studio."
+            )
+
+        config_path = self.config.files["global"]
+        ui.write(f"Authentication complete. Saved token to {config_path}.")
         return 0
 
     def save_config(self, hostname, token):
         with self.config.edit("global") as conf:
             conf["studio"]["token"] = token
             conf["studio"]["url"] = hostname
+            if "auto_push" not in conf["exp"]:
+                conf["exp"]["auto_push"] = True
 
 
 class CmdStudioLogout(CmdConfig):
@@ -54,12 +77,14 @@ class CmdStudioLogout(CmdConfig):
 
         with self.config.edit("global") as conf:
             if not get_in(conf, ["studio", "token"]):
-                ui.error_write("Not logged in to Studio.")
+                ui.error_write(
+                    "Not logged in to Studio. Log in with 'dvc studio login'."
+                )
                 return 1
 
             del conf["studio"]["token"]
 
-        ui.write("Logged out from Studio")
+        ui.write("Logged out from Studio (you can log back in with 'dvc studio login')")
         return 0
 
 

dvc/commands/update.py

@@ -74,7 +74,7 @@ def add_parser(subparsers, parent_parser):
         "--remote",
         help="Remote storage to perform updates to",
         metavar="<name>",
-    )
+    ).complete = completion.REMOTE
     update_parser.add_argument(
         "-j",
         "--jobs",

dvc/compare.py

@@ -1,23 +1,15 @@
 from collections import abc
-from itertools import chain, repeat, zip_longest
-from operator import itemgetter
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Dict,
+from collections.abc import (
     ItemsView,
     Iterable,
     Iterator,
-    List,
     Mapping,
     MutableSequence,
-    Optional,
     Sequence,
-    Set,
-    Tuple,
-    Union,
-    overload,
 )
+from itertools import chain, repeat, zip_longest
+from operator import itemgetter
+from typing import TYPE_CHECKING, Any, Optional, Union, overload
 
 from funcy import reraise
 
@@ -25,7 +17,7 @@ if TYPE_CHECKING:
     from dvc.ui.table import CellT
 
 
-class Column(List["CellT"]):
+class Column(list["CellT"]):
     pass
 
 
@@ -35,13 +27,13 @@ def with_value(value, default):
 
 class TabularData(MutableSequence[Sequence["CellT"]]):
     def __init__(self, columns: Sequence[str], fill_value: Optional[str] = ""):
-        self._columns: Dict[str, Column] = {name: Column() for name in columns}
-        self._keys: List[str] = list(columns)
+        self._columns: dict[str, Column] = {name: Column() for name in columns}
+        self._keys: list[str] = list(columns)
         self._fill_value = fill_value
-        self._protected: Set[str] = set()
+        self._protected: set[str] = set()
 
     @property
-    def columns(self) -> List[Column]:
+    def columns(self) -> list[Column]:
         return list(map(self.column, self.keys()))
 
     def is_protected(self, col_name) -> bool:
@@ -60,10 +52,10 @@ class TabularData(MutableSequence[Sequence["CellT"]]):
         projection = {k: self.column(k) for k in self.keys()}
         return projection.items()
 
-    def keys(self) -> List[str]:
+    def keys(self) -> list[str]:
         return self._keys
 
-    def _iter_col_row(self, row: Sequence["CellT"]) -> Iterator[Tuple["CellT", Column]]:
+    def _iter_col_row(self, row: Sequence["CellT"]) -> Iterator[tuple["CellT", Column]]:
         for val, col in zip_longest(row, self.columns):
             if col is None:
                 break
@@ -81,7 +73,7 @@ class TabularData(MutableSequence[Sequence["CellT"]]):
         for val, col in self._iter_col_row(value):
             col.insert(index, val)
 
-    def __iter__(self) -> Iterator[List["CellT"]]:
+    def __iter__(self) -> Iterator[list["CellT"]]:
         return map(list, zip(*self.columns))
 
     def __getattr__(self, item: str) -> Column:
@@ -96,12 +88,10 @@ class TabularData(MutableSequence[Sequence["CellT"]]):
         return list(it)
 
     @overload
-    def __setitem__(self, item: int, value: Sequence["CellT"]) -> None:
-        ...
+    def __setitem__(self, item: int, value: Sequence["CellT"]) -> None: ...
 
     @overload
-    def __setitem__(self, item: slice, value: Iterable[Sequence["CellT"]]) -> None:
-        ...
+    def __setitem__(self, item: slice, value: Iterable[Sequence["CellT"]]) -> None: ...
 
     def __setitem__(self, item, value) -> None:
         it = value
@@ -127,7 +117,7 @@ class TabularData(MutableSequence[Sequence["CellT"]]):
         return len(self.columns[0])
 
     @property
-    def shape(self) -> Tuple[int, int]:
+    def shape(self) -> tuple[int, int]:
         return len(self.columns), len(self)
 
     def drop(self, *col_names: str) -> None:
@@ -170,7 +160,7 @@ class TabularData(MutableSequence[Sequence["CellT"]]):
             if key not in keys:
                 self.add_column(key)
 
-        row: List["CellT"] = [
+        row: list[CellT] = [
             with_value(d.get(key), self._fill_value) for key in self.keys()
         ]
         self.append(row)
@@ -185,7 +175,7 @@ class TabularData(MutableSequence[Sequence["CellT"]]):
 
     def as_dict(
         self, cols: Optional[Iterable[str]] = None
-    ) -> Iterable[Dict[str, "CellT"]]:
+    ) -> Iterable[dict[str, "CellT"]]:
         keys = self.keys() if cols is None else set(cols)
         return [{k: self._columns[k][i] for k in keys} for i in range(len(self))]
 
@@ -202,7 +192,7 @@ class TabularData(MutableSequence[Sequence["CellT"]]):
         if how not in ["any", "all"]:
             raise ValueError(f"Invalid 'how' value {how}. Choose one of ['any', 'all']")
 
-        match_line: Set = set()
+        match_line: set = set()
         match_any = True
         if how == "all":
             match_any = False
@@ -245,7 +235,7 @@ class TabularData(MutableSequence[Sequence["CellT"]]):
             )
 
         if axis == "cols":
-            cols_to_drop: List[str] = []
+            cols_to_drop: list[str] = []
             for n_col, col in enumerate(self.columns):
                 if subset and self.keys()[n_col] not in subset:
                     continue
@@ -259,7 +249,7 @@ class TabularData(MutableSequence[Sequence["CellT"]]):
 
         elif axis == "rows":
             unique_rows = []
-            rows_to_drop: List[int] = []
+            rows_to_drop: list[int] = []
             for n_row, row in enumerate(self):
                 if subset:
                     row = [
@@ -319,7 +309,7 @@ def diff_table(
 ) -> TabularData:
     a_rev = a_rev or "HEAD"
     b_rev = b_rev or "workspace"
-    headers: List[str] = ["Path", title, a_rev, b_rev, "Change"]
+    headers: list[str] = ["Path", title, a_rev, b_rev, "Change"]
     fill_value = "-"
     td = TabularData(headers, fill_value=fill_value)
 
@@ -394,7 +384,7 @@ def metrics_table(
 
     for branch, val in metrics.items():
         for fname, metric in val.get("data", {}).items():
-            row_data: Dict[str, str] = {"Revision": branch, "Path": fname}
+            row_data: dict[str, str] = {"Revision": branch, "Path": fname}
             metric = metric.get("data", {})
             flattened = (
                 flatten(format_dict(metric))

dvc/config.py

@@ -1,9 +1,12 @@
 """DVC config objects."""
+
+import ntpath
 import os
+import posixpath
 import re
 from contextlib import contextmanager
 from functools import partial
-from typing import TYPE_CHECKING, Dict, Optional
+from typing import TYPE_CHECKING, Optional
 
 from funcy import compact, memoize, re_find
 
@@ -38,18 +41,6 @@ class RemoteNotFoundError(RemoteConfigError):
     pass
 
 
-class MachineConfigError(ConfigError):
-    pass
-
-
-class NoMachineError(MachineConfigError):
-    pass
-
-
-class MachineNotFoundError(MachineConfigError):
-    pass
-
-
 @memoize
 def get_compiled_schema():
     from voluptuous import Schema
@@ -136,9 +127,10 @@ class Config(dict):
             return global_config_dir()
         if level == "system":
             return system_config_dir()
+        return None
 
     @cached_property
-    def files(self) -> Dict[str, str]:
+    def files(self) -> dict[str, str]:
         files = {
             level: os.path.join(self.get_dir(level), self.CONFIG)
             for level in ("system", "global")
@@ -267,12 +259,17 @@ class Config(dict):
         if re.match(r"\w+://", path):
             return path
 
+        if os.name == "nt" and posixpath.isabs(path) and ntpath.sep not in path:
+            return path
+
         if os.path.isabs(path):
             return path
 
         # on windows convert slashes to backslashes
         # to have path compatible with abs_conf_dir
         if os.path.sep == "\\" and "/" in path:
+            if path.startswith("/"):
+                path = path.replace("/", "\\\\", 1)
             path = path.replace("/", "\\")
 
         expanded = os.path.expanduser(path)
@@ -304,6 +301,9 @@ class Config(dict):
         if os.path.expanduser(path) != path:
             return localfs.as_posix(path)
 
+        if os.name == "nt" and posixpath.isabs(path) and ntpath.sep not in path:
+            return path
+
         if isinstance(path, RelPath) or not os.path.isabs(path):
             path = relpath(path, conf_dir)
             return localfs.as_posix(path)
@@ -329,6 +329,7 @@ class Config(dict):
                     "gdrive_user_credentials_file": func,
                     "gdrive_service_account_json_file_path": func,
                     "credentialpath": func,
+                    "configpath": func,
                     "keyfile": func,
                     "cert_path": func,
                     "key_path": func,
@@ -344,7 +345,7 @@ class Config(dict):
         return Schema(dirs_schema, extra=ALLOW_EXTRA)(conf)
 
     def load_config_to_level(self, level=None):
-        merged_conf: Dict = {}
+        merged_conf: dict = {}
         for merge_level in self.LEVELS:
             if merge_level == level:
                 break
@@ -390,7 +391,7 @@ class Config(dict):
 
 
 def _parse_named(conf):
-    result: Dict[str, Dict] = {"remote": {}, "machine": {}, "db": {}}
+    result: dict[str, dict] = {"remote": {}, "machine": {}, "db": {}}
 
     for section, val in conf.items():
         match = re_find(r'^\s*(remote|machine|db)\s*"(.*)"\s*$', section)

dvc/config_schema.py

@@ -1,4 +1,5 @@
 import os
+from typing import TYPE_CHECKING
 from urllib.parse import urlparse
 
 from funcy import once, walk_values
@@ -10,6 +11,7 @@ from voluptuous import (
     Exclusive,
     Invalid,
     Lower,
+    Marker,
     Optional,
     Range,
     Schema,
@@ -17,6 +19,9 @@ from voluptuous import (
 
 from dvc.log import logger
 
+if TYPE_CHECKING:
+    from collections.abc import Iterator
+
 logger = logger.getChild(__name__)
 
 Bool = All(
@@ -112,17 +117,19 @@ class FeatureSchema(Schema):
         return ret
 
 
+DEPRECATED = "==DEPRECATED=="
+
 REMOTE_COMMON = {
     "url": str,
     "checksum_jobs": All(Coerce(int), Range(1)),
     "jobs": All(Coerce(int), Range(1)),
     Optional("worktree"): Bool,
-    Optional("no_traverse"): Bool,  # obsoleted
+    Optional("no_traverse", description=DEPRECATED): Bool,  # obsoleted
     Optional("version_aware"): Bool,
 }
 LOCAL_COMMON = {
     "type": supported_cache_type,
-    Optional("protected", default=False): Bool,  # obsoleted
+    Optional("protected", default=False, description=DEPRECATED): Bool,  # obsoleted
     "shared": All(Lower, Choices("group")),
     Optional("slow_link_warning", default=True): Bool,
     Optional("verify", default=False): Bool,
@@ -144,6 +151,7 @@ WEBDAV_COMMON = {
     "password": str,
     "ask_password": Bool,
     "token": str,
+    "bearer_token_command": str,
     "custom_auth_header": str,
     "cert_path": str,
     "key_path": str,
@@ -152,6 +160,127 @@ WEBDAV_COMMON = {
     Optional("verify", default=False): Bool,
 }
 
+REMOTE_SCHEMAS = {
+    "": LOCAL_COMMON | REMOTE_COMMON,
+    "s3": {
+        "region": str,
+        "profile": str,
+        "credentialpath": str,
+        "configpath": str,
+        "endpointurl": str,
+        "access_key_id": str,
+        "secret_access_key": str,
+        "session_token": str,
+        Optional(
+            "listobjects", default=False, description=DEPRECATED
+        ): Bool,  # obsoleted
+        Optional("use_ssl", default=True): Bool,
+        Optional("allow_anonymous_login", default=False): Bool,
+        "ssl_verify": Any(Bool, str),
+        "sse": str,
+        "sse_kms_key_id": str,
+        "sse_customer_algorithm": str,
+        "sse_customer_key": str,
+        "acl": str,
+        "grant_read": str,
+        "grant_read_acp": str,
+        "grant_write_acp": str,
+        "grant_full_control": str,
+        "cache_regions": bool,
+        "read_timeout": Coerce(int),
+        "connect_timeout": Coerce(int),
+        Optional("verify", default=False): Bool,
+        **REMOTE_COMMON,
+    },
+    "gs": {
+        "projectname": str,
+        "credentialpath": str,
+        "endpointurl": str,
+        Optional("verify", default=False): Bool,
+        Optional("allow_anonymous_login", default=False): Bool,
+        **REMOTE_COMMON,
+    },
+    "ssh": {
+        "type": supported_cache_type,
+        "port": Coerce(int),
+        "user": str,
+        "password": str,
+        "ask_password": Bool,
+        "passphrase": str,
+        "ask_passphrase": Bool,
+        "keyfile": str,
+        "timeout": Coerce(int),
+        "gss_auth": Bool,
+        "allow_agent": Bool,
+        "max_sessions": Coerce(int),
+        Optional("verify", default=False): Bool,
+        **REMOTE_COMMON,
+    },
+    "hdfs": {
+        "user": str,
+        "kerb_ticket": str,
+        "replication": int,
+        **REMOTE_COMMON,
+    },
+    "webhdfs": {
+        "kerberos": Bool,
+        "kerberos_principal": str,
+        "proxy_to": str,
+        "ssl_verify": Any(Bool, str),
+        "token": str,
+        "use_https": Bool,
+        "user": str,
+        "password": str,
+        "data_proxy_target": str,
+        Optional("verify", default=False): Bool,
+        **REMOTE_COMMON,
+    },
+    "azure": {
+        "connection_string": str,
+        "sas_token": str,
+        "account_name": str,
+        "account_key": str,
+        "tenant_id": str,
+        "client_id": str,
+        "client_secret": str,
+        "allow_anonymous_login": Bool,
+        "exclude_environment_credential": Bool,
+        "exclude_visual_studio_code_credential": Bool,
+        "exclude_shared_token_cache_credential": Bool,
+        "exclude_managed_identity_credential": Bool,
+        Optional("verify", default=False): Bool,
+        "timeout": Coerce(int),
+        "read_timeout": Coerce(int),
+        "connection_timeout": Coerce(int),
+        **REMOTE_COMMON,
+    },
+    "oss": {
+        "oss_key_id": str,
+        "oss_key_secret": str,
+        "oss_endpoint": str,
+        Optional("verify", default=True): Bool,
+        **REMOTE_COMMON,
+    },
+    "gdrive": {
+        "profile": str,
+        "gdrive_use_service_account": Bool,
+        "gdrive_client_id": str,
+        "gdrive_client_secret": str,
+        "gdrive_user_credentials_file": str,
+        "gdrive_service_account_user_email": str,
+        "gdrive_service_account_json_file_path": str,
+        Optional("gdrive_trash_only", default=False): Bool,
+        Optional("gdrive_acknowledge_abuse", default=False): Bool,
+        Optional("verify", default=True): Bool,
+        **REMOTE_COMMON,
+    },
+    "http": HTTP_COMMON | REMOTE_COMMON,
+    "https": HTTP_COMMON | REMOTE_COMMON,
+    "webdav": WEBDAV_COMMON | REMOTE_COMMON,
+    "webdavs": WEBDAV_COMMON | REMOTE_COMMON,
+    "remote": {str: object},  # Any of the above options are valid
+}
+
 SCHEMA = {
     "core": {
         "remote": Lower,
@@ -161,145 +290,37 @@ SCHEMA = {
         Optional("hardlink_lock", default=False): Bool,
         Optional("no_scm", default=False): Bool,
         Optional("autostage", default=False): Bool,
-        Optional("experiments"): Bool,  # obsoleted
+        Optional("experiments", description=DEPRECATED): Bool,  # obsoleted
         Optional("check_update", default=True): Bool,
         "site_cache_dir": str,
         "machine": Lower,
     },
     "cache": {
-        "local": str,  # obsoleted
-        "s3": str,  # obsoleted
-        "gs": str,  # obsoleted
-        "hdfs": str,  # obsoleted
-        "webhdfs": str,  # obsoleted
-        "ssh": str,  # obsoleted
-        "azure": str,  # obsoleted
+        Marker("local", description=DEPRECATED): str,  # obsoleted
+        Marker("s3", description=DEPRECATED): str,  # obsoleted
+        Marker("gs", description=DEPRECATED): str,  # obsoleted
+        Marker("hdfs", description=DEPRECATED): str,  # obsoleted
+        Marker("webhdfs", description=DEPRECATED): str,  # obsoleted
+        Marker("ssh", description=DEPRECATED): str,  # obsoleted
+        Marker("azure", description=DEPRECATED): str,  # obsoleted
         # This is for default local cache
         "dir": str,
         **LOCAL_COMMON,
     },
     "remote": {
-        str: ByUrl(
-            {
-                "": {**LOCAL_COMMON, **REMOTE_COMMON},
-                "s3": {
-                    "region": str,
-                    "profile": str,
-                    "credentialpath": str,
-                    "configpath": str,
-                    "endpointurl": str,
-                    "access_key_id": str,
-                    "secret_access_key": str,
-                    "session_token": str,
-                    Optional("listobjects", default=False): Bool,  # obsoleted
-                    Optional("use_ssl", default=True): Bool,
-                    "ssl_verify": Any(Bool, str),
-                    "sse": str,
-                    "sse_kms_key_id": str,
-                    "sse_customer_algorithm": str,
-                    "sse_customer_key": str,
-                    "acl": str,
-                    "grant_read": str,
-                    "grant_read_acp": str,
-                    "grant_write_acp": str,
-                    "grant_full_control": str,
-                    "cache_regions": bool,
-                    "read_timeout": Coerce(int),
-                    "connect_timeout": Coerce(int),
-                    Optional("verify", default=False): Bool,
-                    **REMOTE_COMMON,
-                },
-                "gs": {
-                    "projectname": str,
-                    "credentialpath": str,
-                    "endpointurl": str,
-                    Optional("verify", default=False): Bool,
-                    **REMOTE_COMMON,
-                },
-                "ssh": {
-                    "type": supported_cache_type,
-                    "port": Coerce(int),
-                    "user": str,
-                    "password": str,
-                    "ask_password": Bool,
-                    "passphrase": str,
-                    "ask_passphrase": Bool,
-                    "keyfile": str,
-                    "timeout": Coerce(int),
-                    "gss_auth": Bool,
-                    "allow_agent": Bool,
-                    "max_sessions": Coerce(int),
-                    Optional("verify", default=False): Bool,
-                    **REMOTE_COMMON,
-                },
-                "hdfs": {"user": str, "kerb_ticket": str, **REMOTE_COMMON},
-                "webhdfs": {
-                    "kerberos": Bool,
-                    "kerberos_principal": str,
-                    "proxy_to": str,
-                    "ssl_verify": Any(Bool, str),
-                    "token": str,
-                    "use_https": Bool,
-                    "user": str,
-                    "password": str,
-                    "data_proxy_target": str,
-                    Optional("verify", default=False): Bool,
-                    **REMOTE_COMMON,
-                },
-                "azure": {
-                    "connection_string": str,
-                    "sas_token": str,
-                    "account_name": str,
-                    "account_key": str,
-                    "tenant_id": str,
-                    "client_id": str,
-                    "client_secret": str,
-                    "allow_anonymous_login": Bool,
-                    "exclude_environment_credential": Bool,
-                    "exclude_visual_studio_code_credential": Bool,
-                    "exclude_shared_token_cache_credential": Bool,
-                    "exclude_managed_identity_credential": Bool,
-                    Optional("verify", default=False): Bool,
-                    "timeout": Coerce(int),
-                    "read_timeout": Coerce(int),
-                    "connection_timeout": Coerce(int),
-                    **REMOTE_COMMON,
-                },
-                "oss": {
-                    "oss_key_id": str,
-                    "oss_key_secret": str,
-                    "oss_endpoint": str,
-                    Optional("verify", default=True): Bool,
-                    **REMOTE_COMMON,
-                },
-                "gdrive": {
-                    "profile": str,
-                    "gdrive_use_service_account": Bool,
-                    "gdrive_client_id": str,
-                    "gdrive_client_secret": str,
-                    "gdrive_user_credentials_file": str,
-                    "gdrive_service_account_user_email": str,
-                    "gdrive_service_account_json_file_path": str,
-                    Optional("gdrive_trash_only", default=False): Bool,
-                    Optional("gdrive_acknowledge_abuse", default=False): Bool,
-                    Optional("verify", default=True): Bool,
-                    **REMOTE_COMMON,
-                },
-                "http": {**HTTP_COMMON, **REMOTE_COMMON},
-                "https": {**HTTP_COMMON, **REMOTE_COMMON},
-                "webdav": {**WEBDAV_COMMON, **REMOTE_COMMON},
-                "webdavs": {**WEBDAV_COMMON, **REMOTE_COMMON},
-                "remote": {str: object},  # Any of the above options are valid
-            }
-        )
+        str: ByUrl(REMOTE_SCHEMAS),
     },
     "state": {
-        "dir": str,  # obsoleted
-        "row_limit": All(Coerce(int), Range(1)),  # obsoleted
-        "row_cleanup_quota": All(Coerce(int), Range(0, 100)),  # obsoleted
+        Marker("dir", description=DEPRECATED): str,  # obsoleted
+        Marker("row_limit", description=DEPRECATED): All(
+            Coerce(int), Range(1)
+        ),  # obsoleted
+        Marker("row_cleanup_quota", description=DEPRECATED): All(
+            Coerce(int), Range(0, 100)
+        ),  # obsoleted
     },
     "index": {
-        "dir": str,  # obsoleted
+        Marker("dir", description=DEPRECATED): str,  # obsoleted
     },
     "machine": {
         str: {
@@ -329,13 +350,15 @@ SCHEMA = {
         "out_dir": str,
     },
     "exp": {
-        "code": str,
-        "data": str,
-        "models": str,
-        "metrics": str,
-        "params": str,
-        "plots": str,
-        "live": str,
+        Marker("code", description=DEPRECATED): str,
+        Marker("data", description=DEPRECATED): str,
+        Marker("models", description=DEPRECATED): str,
+        Marker("metrics", description=DEPRECATED): str,
+        Marker("params", description=DEPRECATED): str,
+        Marker("plots", description=DEPRECATED): str,
+        Marker("live", description=DEPRECATED): str,
+        "auto_push": Bool,
+        "git_remote": str,
     },
     "parsing": {
         "bool": All(Lower, Choices("store_true", "boolean_optional")),
@@ -346,6 +369,7 @@ SCHEMA = {
         Exclusive("config_dir", "config_source"): str,
         Exclusive("config_module", "config_source"): str,
         "config_name": str,
+        "plugins_path": str,
     },
     "studio": {
         "token": str,
@@ -361,3 +385,21 @@ SCHEMA = {
         },
     },
 }
+
+
+def config_vars_for_completion(d: dict = SCHEMA, path: str = "") -> "Iterator[str]":
+    for k, v in d.items():
+        if k in ("machine", "feature"):
+            continue
+        if isinstance(k, Marker):
+            if k.description == DEPRECATED:
+                continue
+            k = k.schema
+        if not isinstance(k, str):
+            continue
+
+        keypath = path + k
+        if isinstance(v, dict):
+            yield from config_vars_for_completion(v, keypath + ".")
+        else:
+            yield keypath

dvc/daemon.py

@@ -5,13 +5,14 @@ import logging
 import os
 import subprocess
 import sys
+from collections.abc import Mapping, Sequence
 from contextlib import nullcontext
-from typing import TYPE_CHECKING, Any, List, Mapping, Optional, Sequence, Union
+from typing import TYPE_CHECKING, Any, Optional, Union
 
 from dvc.log import logger
 
 if TYPE_CHECKING:
-    from typing import ContextManager
+    from contextlib import AbstractContextManager
 
 from dvc.env import DVC_DAEMON, DVC_DAEMON_LOGFILE
 from dvc.utils import fix_env, is_binary
@@ -31,10 +32,10 @@ def _win_detached_subprocess(args: Sequence[str], **kwargs) -> int:
     assert os.name == "nt"
 
     from subprocess import (  # type: ignore[attr-defined]
-        CREATE_NEW_PROCESS_GROUP,
-        CREATE_NO_WINDOW,
-        STARTF_USESHOWWINDOW,
-        STARTUPINFO,
+        CREATE_NEW_PROCESS_GROUP,  # ty: ignore[unresolved-import]
+        CREATE_NO_WINDOW,  # ty: ignore[unresolved-import]
+        STARTF_USESHOWWINDOW,  # ty: ignore[unresolved-import]
+        STARTUPINFO,  # ty: ignore[unresolved-import]
     )
 
     # https://stackoverflow.com/a/7006424
@@ -43,10 +44,10 @@ def _win_detached_subprocess(args: Sequence[str], **kwargs) -> int:
 
     startupinfo = STARTUPINFO()
     startupinfo.dwFlags |= STARTF_USESHOWWINDOW
-    popen = subprocess.Popen(
+    popen = subprocess.Popen(  # noqa: S603
         args,
         close_fds=True,
-        shell=False,  # noqa: S603
+        shell=False,
         startupinfo=startupinfo,
         creationflags=creationflags,
         **kwargs,
@@ -55,7 +56,7 @@ def _win_detached_subprocess(args: Sequence[str], **kwargs) -> int:
     return popen.pid
 
 
-def _get_dvc_args() -> List[str]:
+def _get_dvc_args() -> list[str]:
     args = [sys.executable]
     if not is_binary():
         root_dir = os.path.abspath(os.path.dirname(__file__))
@@ -106,12 +107,7 @@ def _posix_detached_subprocess(args: Sequence[str], **kwargs) -> int:
         os.close(read_end)
         return int(pid_str)
 
-    proc = subprocess.Popen(
-        args,
-        shell=False,  # noqa: S603
-        close_fds=True,
-        **kwargs,
-    )
+    proc = subprocess.Popen(args, shell=False, close_fds=True, **kwargs)  # noqa: S603
     os.close(read_end)
     os.write(write_end, str(proc.pid).encode("utf8"))
     os.close(write_end)
@@ -136,7 +132,7 @@ def _map_log_level_to_flag() -> Optional[str]:
     return flags.get(logger.getEffectiveLevel())
 
 
-def daemon(args: List[str]) -> None:
+def daemon(args: list[str]) -> None:
     """Launch a `dvc daemon` command in a detached process.
 
     Args:
@@ -148,12 +144,12 @@ def daemon(args: List[str]) -> None:
 
 
 def _spawn(
-    args: List[str],
-    executable: Optional[Union[str, List[str]]] = None,
+    args: list[str],
+    executable: Optional[Union[str, list[str]]] = None,
     env: Optional[Mapping[str, str]] = None,
     output_file: Optional[str] = None,
 ) -> int:
-    file: "ContextManager[Any]" = nullcontext()
+    file: AbstractContextManager[Any] = nullcontext()
     kwargs = {}
     if output_file:
         file = open(output_file, "ab")  # noqa: SIM115
@@ -168,9 +164,9 @@ def _spawn(
         return _detached_subprocess(executable + args, env=env, **kwargs)
 
 
-def daemonize(args: List[str], executable: Union[None, str, List[str]] = None) -> None:
+def daemonize(args: list[str], executable: Union[str, list[str], None] = None) -> None:
     if os.name not in ("posix", "nt"):
-        return None
+        return
 
     if os.environ.get(DVC_DAEMON):
         logger.debug("skipping launching a new daemon.")

dvc/dagascii.py

@@ -104,21 +104,21 @@ class AsciiCanvas:
                 if dx == 0:
                     y = y0
                 else:
-                    y = y0 + int(round((x - x0) * dy / float(dx)))
+                    y = y0 + round((x - x0) * dy / float(dx))
                 self.point(x, y, char)
         elif y0 < y1:
             for y in range(y0, y1 + 1):
                 if dy == 0:
                     x = x0
                 else:
-                    x = x0 + int(round((y - y0) * dx / float(dy)))
+                    x = x0 + round((y - y0) * dx / float(dy))
                 self.point(x, y, char)
         else:
             for y in range(y1, y0 + 1):
                 if dy == 0:
                     x = x0
                 else:
-                    x = x1 + int(round((y - y1) * dx / float(dy)))
+                    x = x1 + round((y - y1) * dx / float(dy))
                 self.point(x, y, char)
 
     def text(self, x, y, text):
@@ -195,7 +195,7 @@ def _build_sugiyama_layout(vertices, edges):
 
     sug.yspace = VertexViewer.HEIGHT
     sug.xspace = minw
-    sug.route_edge = route_with_lines
+    sug.route_edge = route_with_lines  # ty: ignore[unresolved-attribute]
 
     sug.draw()
 
@@ -258,8 +258,8 @@ def draw(vertices, edges):
     maxx = max(Xs)
     maxy = max(Ys)
 
-    canvas_cols = int(math.ceil(math.ceil(maxx) - math.floor(minx))) + 1
-    canvas_lines = int(round(maxy - miny))
+    canvas_cols = math.ceil(math.ceil(maxx) - math.floor(minx)) + 1
+    canvas_lines = round(maxy - miny)
 
     canvas = AsciiCanvas(canvas_cols, canvas_lines)
 
@@ -270,10 +270,10 @@ def draw(vertices, edges):
             start = edge.view._pts[index - 1]
             end = edge.view._pts[index]
 
-            start_x = int(round(start[0] - minx))
-            start_y = int(round(start[1] - miny))
-            end_x = int(round(end[0] - minx))
-            end_y = int(round(end[1] - miny))
+            start_x = round(start[0] - minx)
+            start_y = round(start[1] - miny)
+            end_x = round(end[0] - minx)
+            end_y = round(end[1] - miny)
 
             assert start_x >= 0
             assert start_y >= 0
@@ -288,12 +288,12 @@ def draw(vertices, edges):
         y = vertex.view.xy[1]
 
         canvas.box(
-            int(round(x - minx)),
-            int(round(y - miny)),
+            round(x - minx),
+            round(y - miny),
             vertex.view.w,
             vertex.view.h,
         )
 
-        canvas.text(int(round(x - minx)) + 1, int(round(y - miny)) + 1, vertex.data)
+        canvas.text(round(x - minx) + 1, round(y - miny) + 1, vertex.data)
 
     return canvas.draw()

dvc/data_cloud.py

@@ -1,6 +1,7 @@
 """Manages dvc remotes that user can use with push/pull/status commands."""
 
-from typing import TYPE_CHECKING, Iterable, Optional, Set, Tuple
+from collections.abc import Iterable
+from typing import TYPE_CHECKING, Optional
 
 from dvc.config import NoRemoteError, RemoteConfigError
 from dvc.log import logger
@@ -50,7 +51,7 @@ class Remote:
 
 def _split_legacy_hash_infos(
     hash_infos: Iterable["HashInfo"],
-) -> Tuple[Set["HashInfo"], Set["HashInfo"]]:
+) -> tuple[set["HashInfo"], set["HashInfo"]]:
     from dvc.cachemgr import LEGACY_HASH_NAMES
 
     legacy = set()

dvc/database.py

@@ -1,12 +1,13 @@
 import os
+from collections.abc import Iterator
 from contextlib import contextmanager
 from dataclasses import dataclass
 from tempfile import NamedTemporaryFile
-from typing import TYPE_CHECKING, Any, Callable, Dict, Iterator, Optional, Union
+from typing import TYPE_CHECKING, Any, Callable, Optional, Union
 
-from sqlalchemy import create_engine
-from sqlalchemy.engine import make_url as _make_url
-from sqlalchemy.exc import NoSuchModuleError
+from sqlalchemy import create_engine  # type: ignore[import]
+from sqlalchemy.engine import make_url as _make_url  # type: ignore[import]
+from sqlalchemy.exc import NoSuchModuleError  # type: ignore[import]
 
 from dvc import env
 from dvc.exceptions import DvcException
@@ -16,7 +17,7 @@ from dvc.utils import env2bool
 
 if TYPE_CHECKING:
     from sqlalchemy.engine import URL, Connectable, Engine
-    from sqlalchemy.sql.expression import Selectable
+    from sqlalchemy.sql.expression import Selectable  # type: ignore[import]
 
 
 logger = logger.getChild(__name__)
@@ -30,7 +31,7 @@ def make_url(url: Union["URL", str], **kwargs: Any) -> "URL":
     return _make_url(url).set(**kwargs)
 
 
-def url_from_config(config: Union[str, "URL", Dict[str, str]]) -> "URL":
+def url_from_config(config: Union[str, "URL", dict[str, str]]) -> "URL":
     if isinstance(config, dict):
         return make_url(**config)
     return make_url(config)
@@ -83,7 +84,7 @@ class Client:
         except Exception as exc:
             if callable(onerror):
                 onerror()
-            logger.exception(
+            logger.exception(  # noqa: LOG007
                 "Could not connect to the database. "
                 "Check your database credentials and try again.",
                 exc_info=False,
@@ -118,7 +119,7 @@ def handle_error(url: "URL"):
 
 @contextmanager
 def client(
-    url_or_config: Union[str, "URL", Dict[str, str]], **engine_kwargs: Any
+    url_or_config: Union[str, "URL", dict[str, str]], **engine_kwargs: Any
 ) -> Iterator[Client]:
     url = url_from_config(url_or_config)
     echo = env2bool(env.DVC_SQLALCHEMY_ECHO, False)

dvc/dependency/__init__.py

@@ -1,9 +1,11 @@
 from collections import defaultdict
-from typing import Any, Dict, List, Mapping, Set
+from collections.abc import Mapping
+from typing import Any
 
 from dvc.output import ARTIFACT_SCHEMA, DIR_FILES_SCHEMA, Output
 
 from .base import Dependency
+from .dataset import DatasetDependency
 from .db import DbDependency
 from .param import ParamsDependency
 from .repo import RepoDependency
@@ -32,12 +34,14 @@ def _get(stage, p, info, **kwargs):
         return DbDependency(stage, d)
 
     assert p
+    if DatasetDependency.is_dataset(p):
+        return DatasetDependency(stage, p, info)
     if repo:
         return RepoDependency(repo, stage, p, info)
     return Dependency(stage, p, info, **kwargs)
 
 
-def loadd_from(stage, d_list):
+def loadd_from(stage, d_list) -> list[Dependency]:
     ret = []
     for d in d_list:
         p = d.pop(Output.PARAM_PATH, None)
@@ -55,23 +59,15 @@ def loads_from(stage, s_list, erepo=None, fs_config=None, db=None):
     info = {RepoDependency.PARAM_REPO: erepo} if erepo else {}
     if db:
         info.update({"db": db})
-    return [
-        _get(
-            stage,
-            s,
-            info.copy(),
-            fs_config=fs_config,
-        )
-        for s in s_list
-    ]
+    return [_get(stage, s, info.copy(), fs_config=fs_config) for s in s_list]
 
 
-def _merge_params(s_list) -> Dict[str, List[str]]:
+def _merge_params(s_list) -> dict[str, list[str]]:
     d = defaultdict(list)
     default_file = ParamsDependency.DEFAULT_PARAMS_FILE
 
     # figure out completely tracked params file, and ignore specific keys
-    wholly_tracked: Set[str] = set()
+    wholly_tracked: set[str] = set()
     for key in s_list:
         if not isinstance(key, dict):
             continue

dvc/dependency/base.py

@@ -1,5 +1,3 @@
-from typing import Dict, Type
-
 from dvc.exceptions import DvcException
 from dvc.fs import download as fs_download
 from dvc.output import Output
@@ -25,11 +23,11 @@ class DependencyIsStageFileError(DvcException):
 class Dependency(Output):
     IS_DEPENDENCY = True
 
-    DoesNotExistError: Type[DvcException] = DependencyDoesNotExistError
-    IsNotFileOrDirError: Type[DvcException] = DependencyIsNotFileOrDirError
-    IsStageFileError: Type[DvcException] = DependencyIsStageFileError
+    DoesNotExistError: type[DvcException] = DependencyDoesNotExistError
+    IsNotFileOrDirError: type[DvcException] = DependencyIsNotFileOrDirError
+    IsStageFileError: type[DvcException] = DependencyIsStageFileError
 
-    def workspace_status(self) -> Dict[str, str]:
+    def workspace_status(self) -> dict[str, str]:
         if self.fs.version_aware:
             old_fs_path = self.fs_path
             try:
@@ -47,7 +45,7 @@ class Dependency(Output):
             self.fs_path = self.fs.version_path(self.fs_path, self.meta.version_id)
 
     def download(self, to, jobs=None):
-        fs_download(self.fs, self.fs_path, to.fs_path, jobs=jobs)
+        return fs_download(self.fs, self.fs_path, to.fs_path, jobs=jobs)
 
     def save(self):
         super().save()

dvc/dependency/dataset.py

@@ -0,0 +1,84 @@
+from typing import TYPE_CHECKING, Any, ClassVar
+from urllib.parse import urlparse
+
+from funcy import compact, merge
+
+from dvc.exceptions import DvcException
+from dvc_data.hashfile.hash_info import HashInfo
+
+from .db import AbstractDependency
+
+if TYPE_CHECKING:
+    from dvc.stage import Stage
+
+
+class DatasetDependency(AbstractDependency):
+    PARAM_DATASET = "dataset"
+    DATASET_SCHEMA: ClassVar[dict] = {PARAM_DATASET: dict}
+
+    def __init__(self, stage: "Stage", p, info, *args, **kwargs):
+        super().__init__(stage, info, *args, **kwargs)
+        self.def_path = p
+        self.name = urlparse(p).netloc
+        dataset_info = info.get(self.PARAM_DATASET) or {}
+        self.hash_info = HashInfo(self.PARAM_DATASET, dataset_info)  # type: ignore[arg-type]
+        self.hash_name = self.PARAM_DATASET
+
+    def __repr__(self):
+        return f"{self.__class__.__name__}({self.def_path!r})"
+
+    def __str__(self):
+        return self.def_path
+
+    @classmethod
+    def is_dataset(cls, p: str):
+        return urlparse(p).scheme == "ds"
+
+    @property
+    def protocol(self):
+        return None
+
+    def dumpd(self, **kwargs):
+        return compact({self.PARAM_PATH: self.def_path, **self.hash_info.to_dict()})
+
+    def fill_values(self, values=None):
+        """Load params values dynamically."""
+        self.hash_info = HashInfo(
+            self.PARAM_DATASET, merge(self.hash_info.value, values or {})
+        )
+
+    def workspace_status(self):
+        ds = self.repo.datasets[self.name]
+        if not ds.lock:
+            return {str(self): "not in sync"}
+
+        info: dict[str, Any] = self.hash_info.value if self.hash_info else {}  # type: ignore[assignment]
+        lock = self.repo.datasets._lock_from_info(info)
+        if not lock:
+            return {str(self): "new"}
+        if lock != ds.lock:
+            return {str(self): "modified"}
+        return {}
+
+    def status(self):
+        return self.workspace_status()
+
+    def get_hash(self):
+        ds = self.repo.datasets[self.name]
+        if not ds.lock:
+            if ds._invalidated:
+                raise DvcException(
+                    "Dataset information is not in sync. "
+                    f"Run 'dvc ds update {self.name}' to sync."
+                )
+            raise DvcException("Dataset information missing from dvc.lock file")
+        return HashInfo(self.PARAM_DATASET, ds.lock.to_dict())  # type: ignore[arg-type]
+
+    def save(self):
+        self.hash_info = self.get_hash()
+
+    def download(self, to, jobs=None):
+        raise NotImplementedError
+
+    def update(self, rev=None):
+        raise NotImplementedError

dvc/dependency/db.py

@@ -1,5 +1,6 @@
+from collections.abc import Iterator
 from contextlib import contextmanager
-from typing import TYPE_CHECKING, Any, Callable, Dict, Iterator, Optional
+from typing import TYPE_CHECKING, Any, Callable, ClassVar, Optional
 
 from funcy import compact, log_durations
 
@@ -10,6 +11,7 @@ from .base import Dependency
 
 if TYPE_CHECKING:
     from dvc.output import Output
+    from dvc.repo import Repo
     from dvc.stage import Stage
 
 logger = logger.getChild(__name__)
@@ -20,9 +22,12 @@ def download_progress(to: "Output") -> Iterator[Callable[[int], Any]]:
     from dvc.ui import ui
     from dvc.ui._rich_progress import DbDownloadProgress
 
-    with log_durations(logger.debug, f"Saving to {to}"), DbDownloadProgress(
-        console=ui.error_console,
-    ) as progress:
+    with (
+        log_durations(logger.debug, f"Saving to {to}"),
+        DbDownloadProgress(
+            console=ui.error_console,
+        ) as progress,
+    ):
         task = progress.add_task("Saving", total=None, output=to)
         yield lambda n: progress.advance(task, advance=n)
         progress.update(task, description="Saved", total=0)
@@ -31,14 +36,18 @@ def download_progress(to: "Output") -> Iterator[Callable[[int], Any]]:
 class AbstractDependency(Dependency):
     """Dependency without workspace/fs/fs_path"""
 
-    def __init__(self, stage: "Stage", info: Dict[str, Any], *args, **kwargs):
-        self.repo = stage.repo
+    def __init__(self, stage: "Stage", info: dict[str, Any], *args, **kwargs):
+        self.repo: Repo = stage.repo
         self.stage = stage
         self.fs = None
         self.fs_path = None
         self.def_path = None  # type: ignore[assignment]
         self.info = info or {}
 
+    @property
+    def is_in_repo(self):
+        return False
+
 
 class DbDependency(AbstractDependency):
     PARAM_CONNECTION = "connection"
@@ -46,7 +55,7 @@ class DbDependency(AbstractDependency):
     PARAM_QUERY = "query"
     PARAM_TABLE = "table"
     PARAM_FILE_FORMAT = "file_format"
-    DB_SCHEMA = {
+    DB_SCHEMA: ClassVar[dict] = {
         PARAM_DB: {
             PARAM_QUERY: str,
             PARAM_CONNECTION: str,
@@ -57,7 +66,7 @@ class DbDependency(AbstractDependency):
 
     def __init__(self, stage: "Stage", info, *args, **kwargs):
         super().__init__(stage, info, *args, **kwargs)
-        self.db_info: Dict[str, str] = self.info.get(self.PARAM_DB, {})
+        self.db_info: dict[str, str] = self.info.get(self.PARAM_DB, {})
         self.connection = self.db_info.get(self.PARAM_CONNECTION)
 
     @property
@@ -65,8 +74,8 @@ class DbDependency(AbstractDependency):
         return self.db_info.get(self.PARAM_QUERY) or self.db_info.get(self.PARAM_TABLE)
 
     def __repr__(self):
-        return "{}:{}".format(
-            self.__class__.__name__, "".join(f"{k}=={v}" for k, v in self.info.items())
+        return "{}: {}".format(
+            self.__class__.__name__, "".join(f"{k}={v}" for k, v in self.info.items())
         )
 
     def __str__(self):
@@ -110,6 +119,7 @@ class DbDependency(AbstractDependency):
             raise DvcException(f"connection {self.connection} not found in config")
 
         file_format = file_format or self.db_info.get(self.PARAM_FILE_FORMAT, "csv")
+        assert file_format
         with client(config) as db:
             msg = "Testing connection"
             with log_durations(logger.debug, msg), ui.status(msg) as status:

dvc/dependency/param.py

@@ -1,7 +1,7 @@
 import os
 import typing
 from collections import defaultdict
-from typing import TYPE_CHECKING, Any, Dict, List, Optional
+from typing import TYPE_CHECKING, Any, Optional
 
 import dpath
 
@@ -37,7 +37,7 @@ class BadParamFileError(DvcException):
 def read_param_file(
     fs: "FileSystem",
     path: str,
-    key_paths: Optional[List[str]] = None,
+    key_paths: Optional[list[str]] = None,
     flatten: bool = False,
     **load_kwargs,
 ) -> Any:
@@ -76,10 +76,7 @@ class ParamsDependency(Dependency):
         self.params = list(params) if params else []
         hash_info = HashInfo()
         if isinstance(params, dict):
-            hash_info = HashInfo(
-                self.PARAM_PARAMS,
-                params,  # type: ignore[arg-type]
-            )
+            hash_info = HashInfo(self.PARAM_PARAMS, params)  # type: ignore[arg-type]
         repo = repo or stage.repo
         path = path or os.path.join(repo.root_dir, self.DEFAULT_PARAMS_FILE)
         super().__init__(stage, path, repo=repo)
@@ -103,19 +100,17 @@ class ParamsDependency(Dependency):
         for param in self.params:
             if param in values:
                 info[param] = values[param]
-        self.hash_info = HashInfo(
-            self.PARAM_PARAMS,
-            info,  # type: ignore[arg-type]
-        )
+        self.hash_info = HashInfo(self.PARAM_PARAMS, info)  # type: ignore[arg-type]
 
     def read_params(
         self, flatten: bool = True, **kwargs: typing.Any
-    ) -> Dict[str, typing.Any]:
+    ) -> dict[str, typing.Any]:
         try:
             self.validate_filepath()
         except MissingParamsFile:
             return {}
 
+        assert self.repo
         try:
             return read_param_file(
                 self.repo.fs,
@@ -134,7 +129,7 @@ class ParamsDependency(Dependency):
 
         from funcy import ldistinct
 
-        status: Dict[str, Any] = defaultdict(dict)
+        status: dict[str, Any] = defaultdict(dict)
         info = self.hash_info.value if self.hash_info else {}
         assert isinstance(info, dict)
         actual = self.read_params()

dvc/dependency/repo.py

@@ -1,5 +1,5 @@
 from copy import deepcopy
-from typing import TYPE_CHECKING, Any, Dict, Optional, Union
+from typing import TYPE_CHECKING, Any, ClassVar, Optional, Union
 
 import voluptuous as vol
 
@@ -9,7 +9,9 @@ from .base import Dependency
 
 if TYPE_CHECKING:
     from dvc.fs import DVCFileSystem
+    from dvc.output import Output
     from dvc.stage import Stage
+    from dvc_data.hashfile.hash_info import HashInfo
 
 
 class RepoDependency(Dependency):
@@ -20,7 +22,7 @@ class RepoDependency(Dependency):
     PARAM_CONFIG = "config"
     PARAM_REMOTE = "remote"
 
-    REPO_SCHEMA = {
+    REPO_SCHEMA: ClassVar[dict] = {
         PARAM_REPO: {
             vol.Required(PARAM_URL): str,
             PARAM_REV: str,
@@ -30,12 +32,12 @@ class RepoDependency(Dependency):
         }
     }
 
-    def __init__(self, def_repo: Dict[str, Any], stage: "Stage", *args, **kwargs):
+    def __init__(self, def_repo: dict[str, Any], stage: "Stage", *args, **kwargs):
         self.def_repo = def_repo
         super().__init__(stage, *args, **kwargs)
 
         self.fs = self._make_fs()
-        self.fs_path = as_posix(self.def_path)
+        self.fs_path = as_posix(self.fs.normpath(self.def_path))
 
     def _parse_path(self, fs, fs_path):  # noqa: ARG002
         return None
@@ -65,7 +67,7 @@ class RepoDependency(Dependency):
             self.def_repo[self.PARAM_REV_LOCK] = rev
 
     @classmethod
-    def _dump_def_repo(cls, def_repo) -> Dict[str, str]:
+    def _dump_def_repo(cls, def_repo) -> dict[str, str]:
         repo = {cls.PARAM_URL: def_repo[cls.PARAM_URL]}
 
         rev = def_repo.get(cls.PARAM_REV)
@@ -85,12 +87,33 @@ class RepoDependency(Dependency):
             repo[cls.PARAM_REMOTE] = remote
         return repo
 
-    def dumpd(self, **kwargs) -> Dict[str, Union[str, Dict[str, str]]]:
+    def dumpd(self, **kwargs) -> dict[str, Union[str, dict[str, str]]]:
         return {
             self.PARAM_PATH: self.def_path,
             self.PARAM_REPO: self._dump_def_repo(self.def_repo),
         }
 
+    def download(self, to: "Output", jobs: Optional[int] = None):
+        from dvc.fs import LocalFileSystem
+
+        files = super().download(to=to, jobs=jobs)
+        if not isinstance(to.fs, LocalFileSystem):
+            return
+
+        hashes: list[tuple[str, HashInfo, dict[str, Any]]] = []
+        for src_path, dest_path, maybe_info in files:
+            try:
+                info = maybe_info or self.fs.info(src_path)
+                hash_info = info["dvc_info"]["entry"].hash_info
+                dest_info = to.fs.info(dest_path)
+            except (KeyError, AttributeError):
+                # If no hash info found, just keep going and output will be hashed later
+                continue
+            if hash_info:
+                hashes.append((dest_path, hash_info, dest_info))
+        cache = to.cache if to.use_cache else to.local_cache
+        cache.state.save_many(hashes, to.fs)
+
     def update(self, rev: Optional[str] = None):
         if rev:
             self.def_repo[self.PARAM_REV] = rev
@@ -123,11 +146,16 @@ class RepoDependency(Dependency):
         else:
             config = Config.load_file(conf)
 
+        # Setup config to the new DVCFileSystem to use the remote repo, but rely on the
+        # local cache instead of the remote's cache. This avoids re-streaming of data,
+        # but messes up the call to `_get_remote_config()` downstream, which will need
+        # to ignore cache parameters.
+        assert self.repo
         config["cache"] = self.repo.config["cache"]
         config["cache"]["dir"] = self.repo.cache.local_cache_dir
 
         return DVCFileSystem(
-            url=self.def_repo[self.PARAM_URL],
+            repo=self.def_repo[self.PARAM_URL],
             rev=rev or self._get_rev(locked=locked),
             subrepos=True,
             config=config,

dvc/dirs.py

@@ -1,4 +1,5 @@
 import os
+from typing import Optional
 
 import platformdirs
 
@@ -20,7 +21,24 @@ def global_config_dir():
     )
 
 
-def site_cache_dir():
-    return os.getenv(env.DVC_SITE_CACHE_DIR) or platformdirs.site_cache_dir(
-        APPNAME, APPAUTHOR, opinion=True
-    )
+def site_cache_dir(config_site_cache_dir: Optional[str] = None):
+    from platformdirs import PlatformDirs
+    from platformdirs.unix import Unix
+
+    if dvc_site_cache_dir := os.getenv(env.DVC_SITE_CACHE_DIR):
+        return dvc_site_cache_dir
+
+    if config_site_cache_dir is not None:
+        return config_site_cache_dir
+
+    if issubclass(Unix, PlatformDirs):
+        # Return the cache directory shared by users, e.g. `/var/tmp/$appname`
+        # NOTE: platformdirs>=5 changed `site_cache_dir` to return /var/cache/$appname.
+        # as the following path is considered insecure.
+        # For details, see: https://github.com/tox-dev/platformdirs/pull/239
+
+        # FIXME: keeping the old behavior temporarily to avoid dependency conflict.
+        #        In the future, consider migrating to a more secure directory.
+        return f"/var/tmp/{APPNAME}"  # noqa: S108
+
+    return platformdirs.site_cache_dir(APPNAME, APPAUTHOR, opinion=True)

dvc/dvcfile.py

@@ -1,16 +1,6 @@
 import contextlib
 import os
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Callable,
-    Dict,
-    List,
-    Optional,
-    Tuple,
-    TypeVar,
-    Union,
-)
+from typing import TYPE_CHECKING, Any, Callable, ClassVar, Optional, TypeVar, Union
 
 from dvc.exceptions import DvcException
 from dvc.log import logger
@@ -78,10 +68,9 @@ def is_git_ignored(repo, path):
 def check_dvcfile_path(repo, path):
     if not is_valid_filename(path):
         raise StageFileBadNameError(
-            "bad DVC file name '{}'. DVC files should be named "
-            "'{}' or have a '.dvc' suffix (e.g. '{}.dvc').".format(
-                relpath(path), PROJECT_FILE, os.path.basename(path)
-            )
+            f"bad DVC file name '{relpath(path)}'. DVC files should be named "
+            f"'{PROJECT_FILE}' or have a '.dvc' suffix "
+            f"(e.g. '{os.path.basename(path)}.dvc')."
         )
 
     if is_git_ignored(repo, path):
@@ -133,7 +122,7 @@ class FileMixin:
         d, _ = self._load(**kwargs)
         return d
 
-    def _load(self, **kwargs: Any) -> Tuple[Any, str]:
+    def _load(self, **kwargs: Any) -> tuple[Any, str]:
         # it raises the proper exceptions by priority:
         # 1. when the file doesn't exists
         # 2. filename is not a DVC file
@@ -156,7 +145,7 @@ class FileMixin:
 
         return validate(d, cls.SCHEMA, path=fname)  # type: ignore[arg-type]
 
-    def _load_yaml(self, **kwargs: Any) -> Tuple[Any, str]:
+    def _load_yaml(self, **kwargs: Any) -> tuple[Any, str]:
         from dvc.utils import strictyaml
 
         return strictyaml.load(
@@ -173,6 +162,9 @@ class FileMixin:
     def dump(self, stage, **kwargs):
         raise NotImplementedError
 
+    def dump_stages(self, stages, **kwargs):
+        raise NotImplementedError
+
     def merge(self, ancestor, other, allowed=None):
         raise NotImplementedError
 
@@ -181,10 +173,12 @@ class SingleStageFile(FileMixin):
     from dvc.schema import COMPILED_SINGLE_STAGE_SCHEMA as SCHEMA
     from dvc.stage.loader import SingleStageLoader as LOADER  # noqa: N814
 
-    metrics: List[str] = []
-    plots: Any = {}
-    params: List[str] = []
-    artifacts: Dict[str, Optional[Dict[str, Any]]] = {}
+    datasets: ClassVar[list[dict[str, Any]]] = []
+    datasets_lock: ClassVar[list[dict[str, Any]]] = []
+    metrics: ClassVar[list[str]] = []
+    plots: ClassVar[Any] = {}
+    params: ClassVar[list[str]] = []
+    artifacts: ClassVar[dict[str, Optional[dict[str, Any]]]] = {}
 
     @property
     def stage(self) -> "Stage":
@@ -207,6 +201,13 @@ class SingleStageFile(FileMixin):
         dump_yaml(self.path, serialize.to_single_stage_file(stage, **kwargs))
         self.repo.scm_context.track_file(self.relpath)
 
+    def dump_stages(self, stages, **kwargs) -> None:
+        if not stages:
+            return None
+
+        assert len(stages) == 1, "SingleStageFile can only dump one stage."
+        return self.dump(stages[0], **kwargs)
+
     def remove_stage(self, stage):  # noqa: ARG002
         self.remove()
 
@@ -237,44 +238,79 @@ class ProjectFile(FileMixin):
 
     def dump(self, stage, update_pipeline=True, update_lock=True, **kwargs):
         """Dumps given stage appropriately in the dvcfile."""
+        return self.dump_stages(
+            [stage], update_pipeline=update_pipeline, update_lock=update_lock, **kwargs
+        )
+
+    def dump_stages(self, stages, update_pipeline=True, update_lock=True, **kwargs):
         from dvc.stage import PipelineStage
 
-        assert isinstance(stage, PipelineStage)
+        if not stages:
+            return
+
+        for stage in stages:
+            assert isinstance(stage, PipelineStage)
+
         if self.verify:
             check_dvcfile_path(self.repo, self.path)
 
-        if update_pipeline and not stage.is_data_source:
-            self._dump_pipeline_file(stage)
+        if update_pipeline:
+            self._dump_pipeline_file(stages)
 
         if update_lock:
-            self._dump_lockfile(stage, **kwargs)
+            self._dump_lockfile(stages, **kwargs)
 
-    def _dump_lockfile(self, stage, **kwargs):
-        self._lockfile.dump(stage, **kwargs)
+    def dump_dataset(self, dataset):
+        with modify_yaml(self.path, fs=self.repo.fs) as data:
+            parsed = self.datasets if data else []
+            raw = data.setdefault("datasets", [])
+            loc = next(
+                (i for i, ds in enumerate(parsed) if ds["name"] == dataset["name"]),
+                None,
+            )
+            if loc is not None:
+                if raw[loc] != parsed[loc]:
+                    raise ParametrizedDumpError(
+                        "cannot update a parametrized dataset entry"
+                    )
+
+                apply_diff(dataset, raw[loc])
+                raw[loc] = dataset
+            else:
+                raw.append(dataset)
+        self.repo.scm_context.track_file(self.relpath)
+
+    def _dump_lockfile(self, stages, **kwargs):
+        self._lockfile.dump_stages(stages, **kwargs)
 
     @staticmethod
     def _check_if_parametrized(stage, action: str = "dump") -> None:
         if stage.raw_data.parametrized:
             raise ParametrizedDumpError(f"cannot {action} a parametrized {stage}")
 
-    def _dump_pipeline_file(self, stage):
-        self._check_if_parametrized(stage)
-        stage_data = serialize.to_pipeline_file(stage)
+    def _dump_pipeline_file(self, stages):
+        stages = stages if isinstance(stages, list) else [stages]
+        if not stages:
+            return
+
+        for stage in stages:
+            self._check_if_parametrized(stage)
 
         with modify_yaml(self.path, fs=self.repo.fs) as data:
             if not data:
                 logger.info("Creating '%s'", self.relpath)
 
             data["stages"] = data.get("stages", {})
-            existing_entry = stage.name in data["stages"]
-            action = "Modifying" if existing_entry else "Adding"
-            logger.info("%s stage '%s' in '%s'", action, stage.name, self.relpath)
-
-            if existing_entry:
-                orig_stage_data = data["stages"][stage.name]
-                apply_diff(stage_data[stage.name], orig_stage_data)
-            else:
-                data["stages"].update(stage_data)
+            for stage in stages:
+                stage_data = serialize.to_pipeline_file(stage)
+                existing_entry = stage.name in data["stages"]
+                action = "Modifying" if existing_entry else "Adding"
+                logger.info("%s stage '%s' in '%s'", action, stage.name, self.relpath)
+                if existing_entry:
+                    orig_stage_data = data["stages"][stage.name]
+                    apply_diff(stage_data[stage.name], orig_stage_data)
+                else:
+                    data["stages"].update(stage_data)
 
         self.repo.scm_context.track_file(self.relpath)
 
@@ -283,11 +319,11 @@ class ProjectFile(FileMixin):
         raise DvcException("ProjectFile has multiple stages. Please specify it's name.")
 
     @cached_property
-    def contents(self) -> Dict[str, Any]:
+    def contents(self) -> dict[str, Any]:
         return self._load()[0]
 
     @cached_property
-    def lockfile_contents(self) -> Dict[str, Any]:
+    def lockfile_contents(self) -> dict[str, Any]:
         return self._lockfile.load()
 
     @cached_property
@@ -302,20 +338,28 @@ class ProjectFile(FileMixin):
         return self.LOADER(self, self.contents, self.lockfile_contents)
 
     @property
-    def metrics(self) -> List[str]:
-        return self.contents.get("metrics", [])
+    def artifacts(self) -> dict[str, Optional[dict[str, Any]]]:
+        return self.resolver.resolve_artifacts()
 
     @property
-    def plots(self) -> Any:
-        return self.contents.get("plots", {})
+    def metrics(self) -> list[str]:
+        return self.resolver.resolve_metrics()
 
     @property
-    def params(self) -> List[str]:
-        return self.contents.get("params", [])
+    def params(self) -> list[str]:
+        return self.resolver.resolve_params()
 
     @property
-    def artifacts(self) -> Dict[str, Optional[Dict[str, Any]]]:
-        return self.contents.get("artifacts", {})
+    def plots(self) -> list[Any]:
+        return self.resolver.resolve_plots()
+
+    @property
+    def datasets(self) -> list[dict[str, Any]]:
+        return self.resolver.resolve_datasets()
+
+    @property
+    def datasets_lock(self) -> list[dict[str, Any]]:
+        return self.lockfile_contents.get("datasets", [])
 
     def remove(self, force=False):
         if not force:
@@ -362,9 +406,30 @@ class Lockfile(FileMixin):
             self._check_gitignored()
             return {}, ""
 
-    def dump(self, stage, **kwargs):
-        stage_data = serialize.to_lockfile(stage, **kwargs)
+    def dump_dataset(self, dataset: dict):
+        with modify_yaml(self.path, fs=self.repo.fs) as data:
+            data.update({"schema": "2.0"})
+            if not data:
+                logger.info("Generating lock file '%s'", self.relpath)
 
+            datasets: list[dict] = data.setdefault("datasets", [])
+            loc = next(
+                (i for i, ds in enumerate(datasets) if ds["name"] == dataset["name"]),
+                None,
+            )
+            if loc is not None:
+                datasets[loc] = dataset
+            else:
+                datasets.append(dataset)
+            data.setdefault("stages", {})
+        self.repo.scm_context.track_file(self.relpath)
+
+    def dump_stages(self, stages, **kwargs):
+        if not stages:
+            return
+
+        is_modified = False
+        log_updated = False
         with modify_yaml(self.path, fs=self.repo.fs) as data:
             if not data:
                 data.update({"schema": "2.0"})
@@ -372,17 +437,24 @@ class Lockfile(FileMixin):
                 logger.info("Generating lock file '%s'", self.relpath)
 
             data["stages"] = data.get("stages", {})
-            modified = data["stages"].get(stage.name, {}) != stage_data.get(
-                stage.name, {}
-            )
-            if modified:
-                logger.info("Updating lock file '%s'", self.relpath)
+            for stage in stages:
+                stage_data = serialize.to_lockfile(stage, **kwargs)
+                modified = data["stages"].get(stage.name, {}) != stage_data.get(
+                    stage.name, {}
+                )
+                if modified:
+                    is_modified = True
+                    if not log_updated:
+                        logger.info("Updating lock file '%s'", self.relpath)
+                        log_updated = True
+                data["stages"].update(stage_data)
 
-            data["stages"].update(stage_data)
-
-        if modified:
+        if is_modified:
             self.repo.scm_context.track_file(self.relpath)
 
+    def dump(self, stage, **kwargs):
+        self.dump_stages([stage], **kwargs)
+
     def remove_stage(self, stage):
         if not self.exists():
             return

dvc/env.py

@@ -19,3 +19,4 @@ DVC_STUDIO_URL = "DVC_STUDIO_URL"
 DVC_SQLALCHEMY_ECHO = "DVC_SQLALCHEMY_ECHO"
 DVC_SYSTEM_CONFIG_DIR = "DVC_SYSTEM_CONFIG_DIR"
 DVC_UPDATER_ENDPOINT = "DVC_UPDATER_ENDPOINT"
+DVC_STAGE = "DVC_STAGE"

dvc/exceptions.py

@@ -1,6 +1,7 @@
 """Exceptions raised by the dvc."""
+
 import errno
-from typing import TYPE_CHECKING, Dict, List, Optional, Set
+from typing import TYPE_CHECKING, Optional
 
 from dvc.utils import format_link
 
@@ -8,7 +9,7 @@ if TYPE_CHECKING:
     from dvc.stage import Stage
 
 
-class DvcException(Exception):
+class DvcException(Exception):  # noqa: N818
     """Base class for all dvc exceptions."""
 
     def __init__(self, msg, *args):
@@ -34,7 +35,7 @@ class OutputDuplicationError(DvcException):
         stages (list): list of paths to stages.
     """
 
-    def __init__(self, output: str, stages: Set["Stage"]):
+    def __init__(self, output: str, stages: set["Stage"]):
         from funcy import first
 
         assert isinstance(output, str)
@@ -132,11 +133,11 @@ class MoveNotDataSourceError(DvcException):
     def __init__(self, path):
         msg = (
             "move is not permitted for stages that are not data sources. "
-            "You need to either move '{path}' to a new location and edit "
-            "it by hand, or remove '{path}' and create a new one at the "
+            f"You need to either move {path!r} to a new location and edit "
+            f"it by hand, or remove {path!r} and create a new one at the "
             "desired location."
         )
-        super().__init__(msg.format(path=path))
+        super().__init__(msg)
 
 
 class NotDvcRepoError(DvcException):
@@ -241,11 +242,11 @@ class UploadError(FileTransferError):
 
 
 class CheckoutError(DvcException):
-    def __init__(self, target_infos: List[str], stats: Dict[str, List[str]]):
+    def __init__(self, target_infos: list[str], result: dict):
         from dvc.utils import error_link
 
         self.target_infos = target_infos
-        self.stats = stats
+        self.result = result
         targets = [str(t) for t in target_infos]
         m = (
             "Checkout failed for following targets:\n{}\nIs your "
@@ -270,9 +271,8 @@ class NoOutputInExternalRepoError(DvcException):
         from dvc.utils import relpath
 
         super().__init__(
-            "Output '{}' not found in target repository '{}'".format(
-                relpath(path, external_repo_path), external_repo_url
-            )
+            f"Output {relpath(path, external_repo_path)!r} "
+            f"not found in target repository '{external_repo_url}'"
         )
 
 

dvc/fs/__init__.py

@@ -1,10 +1,11 @@
+import glob
+from itertools import repeat
 from typing import Optional
 from urllib.parse import urlparse
 
-from dvc_http import HTTPFileSystem, HTTPSFileSystem  # noqa: F401
-
 from dvc.config import ConfigError as RepoConfigError
 from dvc.config_schema import SCHEMA, Invalid
+from dvc_http import HTTPFileSystem, HTTPSFileSystem  # noqa: F401
 
 # pylint: disable=unused-import
 from dvc_objects.fs import (  # noqa: F401
@@ -19,7 +20,7 @@ from dvc_objects.fs import (  # noqa: F401
     system,
     utils,
 )
-from dvc_objects.fs.base import AnyFSPath, FileSystem  # noqa: F401, TCH001
+from dvc_objects.fs.base import AnyFSPath, FileSystem  # noqa: F401, TC001
 from dvc_objects.fs.errors import (  # noqa: F401
     AuthError,
     ConfigError,
@@ -47,15 +48,24 @@ known_implementations.update(
 
 def download(
     fs: "FileSystem", fs_path: str, to: str, jobs: Optional[int] = None
-) -> int:
+) -> list[tuple[str, str, Optional[dict]]]:
     from dvc.scm import lfs_prefetch
 
     from .callbacks import TqdmCallback
 
-    with TqdmCallback(
-        desc=f"Downloading {fs.name(fs_path)}",
-        unit="files",
-    ) as cb:
+    with TqdmCallback(desc=f"Downloading {fs.name(fs_path)}", unit="files") as cb:
+        if isinstance(fs, DVCFileSystem):
+            lfs_prefetch(
+                fs,
+                [
+                    f"{fs.normpath(glob.escape(fs_path))}/**"
+                    if fs.isdir(fs_path)
+                    else glob.escape(fs_path)
+                ],
+            )
+            if not glob.has_magic(fs_path):
+                return fs._get(fs_path, to, batch_size=jobs, callback=cb)
+
         # NOTE: We use dvc-objects generic.copy over fs.get since it makes file
         # download atomic and avoids fsspec glob/regex path expansion.
         if fs.isdir(fs_path):
@@ -64,7 +74,7 @@ def download(
             ]
             if not from_infos:
                 localfs.makedirs(to, exist_ok=True)
-                return 0
+                return []
             to_infos = [
                 localfs.join(to, *fs.relparts(info, fs_path)) for info in from_infos
             ]
@@ -72,19 +82,10 @@ def download(
             from_infos = [fs_path]
             to_infos = [to]
 
-        if isinstance(fs, DVCFileSystem):
-            lfs_prefetch(fs, from_infos)
         cb.set_size(len(from_infos))
         jobs = jobs or fs.jobs
-        generic.copy(
-            fs,
-            from_infos,
-            localfs,
-            to_infos,
-            callback=cb,
-            batch_size=jobs,
-        )
-        return len(to_infos)
+        generic.copy(fs, from_infos, localfs, to_infos, callback=cb, batch_size=jobs)
+        return list(zip(from_infos, to_infos, repeat(None)))
 
 
 def parse_external_url(url, fs_config=None, config=None):
@@ -168,5 +169,5 @@ def get_cloud_fs(repo_config, **kwargs):
         fs_path = cls._strip_protocol(url)
 
     extras = cls._get_kwargs_from_urls(url)
-    conf = {**extras, **remote_conf}  # remote config takes priority
+    conf = extras | remote_conf  # remote config takes priority
     return cls, conf, fs_path

dvc/fs/callbacks.py

@@ -1,67 +1,38 @@
 from contextlib import ExitStack
-from typing import TYPE_CHECKING, Any, BinaryIO, Dict, Optional, Union
+from typing import TYPE_CHECKING, BinaryIO, Optional, Union
+
+from fsspec.callbacks import DEFAULT_CALLBACK, Callback  # noqa: F401
+from fsspec.callbacks import TqdmCallback as _TqdmCallback
 
 from dvc.progress import Tqdm
 from dvc.utils.objects import cached_property
-from dvc_objects.fs.callbacks import DEFAULT_CALLBACK, Callback  # noqa: F401
 
 if TYPE_CHECKING:
     from rich.progress import TaskID
+    from tqdm import tqdm
 
     from dvc.ui._rich_progress import RichTransferProgress
 
 
-# NOTE: this is very similar to dvc-objects.fs.callbacks.TqdmCallback,
-# but it works with our own Tqdm instance.
-class TqdmCallback(Callback):
+class TqdmCallback(_TqdmCallback):
     def __init__(
         self,
         size: Optional[int] = None,
         value: int = 0,
-        progress_bar: Optional["Tqdm"] = None,
+        progress_bar: Optional["tqdm"] = None,
+        tqdm_cls: Optional[type["tqdm"]] = None,
         **tqdm_kwargs,
     ):
-        tqdm_kwargs["total"] = size or -1
-        self._tqdm_kwargs = tqdm_kwargs
-        self._progress_bar = progress_bar
-        self._stack = ExitStack()
-        super().__init__(size=size, value=value)
-
-    @cached_property
-    def progress_bar(self):
-        progress_bar = (
-            self._progress_bar
-            if self._progress_bar is not None
-            else Tqdm(**self._tqdm_kwargs)
+        tqdm_kwargs.pop("total", None)
+        super().__init__(
+            tqdm_kwargs=tqdm_kwargs, tqdm_cls=tqdm_cls or Tqdm, size=size, value=value
         )
-        return self._stack.enter_context(progress_bar)
+        if progress_bar is not None:
+            self.tqdm = progress_bar
 
-    def __enter__(self):
-        return self
-
-    def close(self):
-        self._stack.close()
-
-    def set_size(self, size):
-        # Tqdm tries to be smart when to refresh,
-        # so we try to force it to re-render.
-        super().set_size(size)
-        self.progress_bar.refresh()
-
-    def call(self, hook_name=None, **kwargs):  # noqa: ARG002
-        self.progress_bar.update_to(self.value, total=self.size)
-
-    def branch(
-        self,
-        path_1: "Union[str, BinaryIO]",
-        path_2: str,
-        kwargs: Dict[str, Any],
-        child: Optional[Callback] = None,
-    ):
-        child = child or TqdmCallback(
-            bytes=True, desc=path_1 if isinstance(path_1, str) else path_2
-        )
-        return super().branch(path_1, path_2, kwargs, child=child)
+    def branched(self, path_1: "Union[str, BinaryIO]", path_2: str, **kwargs):
+        desc = path_1 if isinstance(path_1, str) else path_2
+        return TqdmCallback(bytes=True, desc=desc)
 
 
 class RichCallback(Callback):
@@ -110,9 +81,6 @@ class RichCallback(Callback):
     def task(self) -> "TaskID":
         return self.progress.add_task(**self._task_kwargs)  # type: ignore[arg-type]
 
-    def __enter__(self):
-        return self
-
     def close(self):
         if self._transient:
             self.progress.clear_task(self.task)
@@ -126,17 +94,10 @@ class RichCallback(Callback):
             visible=not self.disable,
         )
 
-    def branch(
-        self,
-        path_1: Union[str, BinaryIO],
-        path_2: str,
-        kwargs: Dict[str, Any],
-        child: Optional["Callback"] = None,
-    ):
-        child = child or RichCallback(
+    def branched(self, path_1: Union[str, BinaryIO], path_2: str, **kwargs):
+        return RichCallback(
             progress=self.progress,
             desc=path_1 if isinstance(path_1, str) else path_2,
             bytes=True,
             transient=self._transient,
         )
-        return super().branch(path_1, path_2, kwargs, child=child)

dvc/fs/data.py

@@ -22,9 +22,7 @@ class DataFileSystem(FileSystem):
         return config
 
     @functools.cached_property
-    def fs(
-        self,
-    ) -> "_DataFileSystem":
+    def fs(self) -> "_DataFileSystem":
         from dvc_data.fs import DataFileSystem as _DataFileSystem
 
         return _DataFileSystem(**self.fs_args)

dvc/fs/dvc.py

@@ -4,26 +4,32 @@ import ntpath
 import os
 import posixpath
 import threading
-from collections import deque
-from contextlib import ExitStack, suppress
-from typing import TYPE_CHECKING, Any, Callable, Dict, Optional, Tuple, Type, Union
+from collections import defaultdict, deque
+from contextlib import ExitStack, nullcontext, suppress
+from glob import has_magic
+from typing import TYPE_CHECKING, Any, Callable, Optional, Union
 
-from fsspec.spec import AbstractFileSystem
+from fsspec.spec import DEFAULT_CALLBACK, AbstractFileSystem
 from funcy import wrap_with
 
 from dvc.log import logger
-from dvc_objects.fs.base import FileSystem
+from dvc.utils.threadpool import ThreadPoolExecutor
+from dvc_objects.fs.base import AnyFSPath, FileSystem
 
 from .data import DataFileSystem
 
 if TYPE_CHECKING:
+    from contextlib import AbstractContextManager
+
     from dvc.repo import Repo
     from dvc.types import DictStrAny, StrPath
 
+    from .callbacks import Callback
+
 logger = logger.getChild(__name__)
 
-RepoFactory = Union[Callable[..., "Repo"], Type["Repo"]]
-Key = Tuple[str, ...]
+RepoFactory = Union[Callable[..., "Repo"], type["Repo"]]
+Key = tuple[str, ...]
 
 
 def as_posix(path: str) -> str:
@@ -77,15 +83,14 @@ class _DVCFileSystem(AbstractFileSystem):
     cachable = False
     root_marker = "/"
 
-    def __init__(  # noqa: PLR0913
+    def __init__(
         self,
-        url: Optional[str] = None,
+        repo: Union["Repo", os.PathLike[str], str, None] = None,
         rev: Optional[str] = None,
-        repo: Optional["Repo"] = None,
         subrepos: bool = False,
         repo_factory: Optional[RepoFactory] = None,
         fo: Optional[str] = None,
-        target_options: Optional[Dict[str, Any]] = None,  # noqa: ARG002
+        target_options: Optional[dict[str, Any]] = None,  # noqa: ARG002
         target_protocol: Optional[str] = None,  # noqa: ARG002
         config: Optional["DictStrAny"] = None,
         remote: Optional[str] = None,
@@ -95,7 +100,8 @@ class _DVCFileSystem(AbstractFileSystem):
         """DVC + git-tracked files fs.
 
         Args:
-            path (str, optional): URL or path to a DVC/Git repository.
+            repo (str | os.PathLike[str] | Repo, optional): A url or a path to a DVC/Git
+                repository, or a `Repo` instance.
                 Defaults to a DVC repository in the current working directory.
                 Both HTTP and SSH protocols are supported for remote Git repos
                 (e.g. [user@]server:project.git).
@@ -105,7 +111,6 @@ class _DVCFileSystem(AbstractFileSystem):
                 In case of a local repository, if rev is unspecified, it will
                 default to the working directory.
                 If the repo is not a Git repo, this option is ignored.
-            repo (:obj:`Repo`, optional): `Repo` instance.
             subrepos (bool): traverse to subrepos.
                 By default, it ignores subrepos.
             repo_factory (callable): A function to initialize subrepo with.
@@ -130,13 +135,23 @@ class _DVCFileSystem(AbstractFileSystem):
             ...    rev="main",
             ... )
         """
+        from dvc.repo import Repo
+
+        # kwargs.get("url") is for maintaining backward compatibility
+        repo = repo or fo or kwargs.get("url")
+        if isinstance(repo, Repo):
+            self._repo: Optional[Repo] = repo
+            url = None
+        else:
+            self._repo = None
+            url = os.fspath(repo) if repo else None
+
         super().__init__()
-        self._repo = repo
         self._repo_factory = repo_factory
         self._traverse_subrepos = subrepos
         self._repo_stack = ExitStack()
         self._repo_kwargs = {
-            "url": url if url is not None else fo,
+            "url": url,
             "rev": rev,
             "subrepos": subrepos,
             "config": config,
@@ -145,7 +160,7 @@ class _DVCFileSystem(AbstractFileSystem):
         }
 
     def getcwd(self):
-        relparts: Tuple[str, ...] = ()
+        relparts: tuple[str, ...] = ()
         assert self.repo is not None
         if self.repo.fs.isin(self.repo.fs.getcwd(), self.repo.root_dir):
             relparts = self.repo.fs.relparts(self.repo.fs.getcwd(), self.repo.root_dir)
@@ -156,7 +171,7 @@ class _DVCFileSystem(AbstractFileSystem):
         return posixpath.join(*parts)
 
     @classmethod
-    def parts(cls, path: str) -> Tuple[str, ...]:
+    def parts(cls, path: str) -> tuple[str, ...]:
         ret = []
         while True:
             path, part = posixpath.split(path)
@@ -187,7 +202,7 @@ class _DVCFileSystem(AbstractFileSystem):
             start = "."
         return posixpath.relpath(self.abspath(path), start=self.abspath(start))
 
-    def relparts(self, path: str, start: Optional[str] = None) -> Tuple[str, ...]:
+    def relparts(self, path: str, start: Optional[str] = None) -> tuple[str, ...]:
         return self.parts(self.relpath(path, start=start))
 
     @functools.cached_property
@@ -321,7 +336,7 @@ class _DVCFileSystem(AbstractFileSystem):
 
     def _get_subrepo_info(
         self, key: Key
-    ) -> Tuple["Repo", Optional[DataFileSystem], Key]:
+    ) -> tuple["Repo", Optional[DataFileSystem], Key]:
         """
         Returns information about the subrepo the key is part of.
         """
@@ -360,38 +375,47 @@ class _DVCFileSystem(AbstractFileSystem):
         except FileNotFoundError:
             return False
 
-    def ls(self, path, detail=True, dvc_only=False, **kwargs):
+    def ls(self, path, detail=True, dvc_only=False, **kwargs):  # noqa: C901, PLR0912
         key = self._get_key_from_relative(path)
         repo, dvc_fs, subkey = self._get_subrepo_info(key)
 
-        dvc_exists = False
         dvc_infos = {}
+        dvc_info = {}
         if dvc_fs:
             dvc_path = _get_dvc_path(dvc_fs, subkey)
-            with suppress(FileNotFoundError, NotADirectoryError):
-                for info in dvc_fs.ls(dvc_path, detail=True):
-                    dvc_infos[dvc_fs.name(info["name"])] = info
-                dvc_exists = True
+            with suppress(FileNotFoundError):
+                dvc_info = dvc_fs.info(dvc_path)
+                if dvc_info["type"] == "file":
+                    dvc_infos[""] = dvc_info
+                else:
+                    for info in dvc_fs.ls(dvc_path, detail=True):
+                        dvc_infos[dvc_fs.name(info["name"])] = info
 
-        fs_exists = False
         fs_infos = {}
+        fs_info = {}
         ignore_subrepos = kwargs.get("ignore_subrepos", True)
         if not dvc_only:
             fs = self.repo.fs
             fs_path = self._from_key(key)
             try:
-                for info in repo.dvcignore.ls(
-                    fs, fs_path, detail=True, ignore_subrepos=ignore_subrepos
-                ):
-                    fs_infos[fs.name(info["name"])] = info
-                fs_exists = True
+                fs_info = fs.info(fs_path)
+                if fs_info["type"] == "file":
+                    fs_infos[""] = fs_info
+                else:
+                    for info in repo.dvcignore.ls(
+                        fs, fs_path, detail=True, ignore_subrepos=ignore_subrepos
+                    ):
+                        fs_infos[fs.name(info["name"])] = info
             except (FileNotFoundError, NotADirectoryError):
                 pass
 
-        if not (dvc_exists or fs_exists):
+        if not (fs_info or dvc_info):
             # broken symlink or TreeError
             raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), path)
 
+        if fs_info and dvc_info and dvc_info["type"] != fs_info["type"]:
+            dvc_infos.clear()  # invalidate dvc_info if file type differs
+
         dvcfiles = kwargs.get("dvcfiles", False)
 
         infos = []
@@ -402,7 +426,7 @@ class _DVCFileSystem(AbstractFileSystem):
             if not dvcfiles and _is_dvc_file(name):
                 continue
 
-            entry_path = self.join(path, name)
+            entry_path = self.join(path, name) if name else path
             info = _merge_info(
                 repo, (*subkey, name), fs_infos.get(name), dvc_infos.get(name)
             )
@@ -420,7 +444,7 @@ class _DVCFileSystem(AbstractFileSystem):
         ignore_subrepos = kwargs.get("ignore_subrepos", True)
         return self._info(key, path, ignore_subrepos=ignore_subrepos)
 
-    def _info(  # noqa: C901, PLR0912
+    def _info(  # noqa: C901
         self, key, path, ignore_subrepos=True, check_ignored=True
     ):
         repo, dvc_fs, subkey = self._get_subrepo_info(key)
@@ -459,15 +483,134 @@ class _DVCFileSystem(AbstractFileSystem):
                     continue
 
         if not dvc_info and not fs_info:
-            raise FileNotFoundError
+            raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), path)
 
         info = _merge_info(repo, subkey, fs_info, dvc_info)
         info["name"] = path
         return info
 
+    def get(
+        self,
+        rpath,
+        lpath,
+        recursive=False,
+        callback=DEFAULT_CALLBACK,
+        maxdepth=None,
+        batch_size=None,
+        **kwargs,
+    ):
+        self._get(
+            rpath,
+            lpath,
+            recursive=recursive,
+            callback=callback,
+            maxdepth=maxdepth,
+            batch_size=batch_size,
+            **kwargs,
+        )
+
+    def _get(  # noqa: C901, PLR0912, PLR0915
+        self,
+        rpath,
+        lpath,
+        recursive=False,
+        callback=DEFAULT_CALLBACK,
+        maxdepth=None,
+        batch_size=None,
+        **kwargs,
+    ) -> list[tuple[str, str, Optional[dict]]]:
+        if (
+            isinstance(rpath, list)
+            or isinstance(lpath, list)
+            or has_magic(rpath)
+            or not self.exists(rpath)
+            or not recursive
+        ):
+            super().get(
+                rpath,
+                lpath,
+                recursive=recursive,
+                callback=callback,
+                maxdepth=maxdepth,
+                **kwargs,
+            )
+            return []
+
+        if os.path.isdir(lpath) or lpath.endswith(os.path.sep):
+            lpath = self.join(lpath, os.path.basename(rpath))
+
+        if self.isfile(rpath):
+            with callback.branched(rpath, lpath) as child:
+                self.get_file(rpath, lpath, callback=child, **kwargs)
+                return [(rpath, lpath, None)]
+
+        result: list[tuple[str, str, Optional[dict]]] = []
+        _dirs: list[str] = []
+        _files: dict[FileSystem, list[tuple[str, str, Optional[dict]]]]
+        _files = defaultdict(list)
+
+        for root, dirs, files in self.walk(rpath, maxdepth=maxdepth, detail=True):
+            if files:
+                callback.set_size((callback.size or 0) + len(files))
+
+            parts = self.relparts(root, rpath)
+            if parts in ((os.curdir,), ("",)):
+                parts = ()
+            dest_root = os.path.join(lpath, *parts)
+            if not maxdepth or len(parts) < maxdepth - 1:
+                _dirs.extend(f"{dest_root}{os.path.sep}{d}" for d in dirs)
+
+            key = self._get_key_from_relative(root)
+            _, dvc_fs, _ = self._get_subrepo_info(key)
+
+            for name, info in files.items():
+                dvc_info = info.get("dvc_info")
+                fs_info = info.get("fs_info")
+                if dvc_fs and dvc_info and not fs_info:
+                    fs = dvc_fs
+                    fs_path = dvc_info["name"]
+                else:
+                    fs = self.repo.fs
+                    fs_path = fs_info["name"]
+
+                src_path = f"{root}{self.sep}{name}"
+                dest_path = f"{dest_root}{os.path.sep}{name}"
+                _files[fs].append((fs_path, dest_path, dvc_info))
+                result.append((src_path, dest_path, info))
+
+        os.makedirs(lpath, exist_ok=True)
+        for d in _dirs:
+            os.makedirs(d, exist_ok=True)
+
+        def get_file(arg: tuple[FileSystem, tuple[str, str, Optional[dict]]]):
+            fs, (src, dest, info) = arg
+            kw = kwargs
+            if isinstance(fs, DataFileSystem):
+                kw = kw | {"info": info}
+            with callback.branched(src, dest) as child:
+                fs.get_file(src, dest, callback=child, **kw)
+
+        if batch_size == 1:
+            ctx: AbstractContextManager = nullcontext()
+            map_fn: Callable = map
+        else:
+            ctx = ThreadPoolExecutor(max_workers=batch_size)
+            map_fn = ctx.imap_unordered
+
+        with ctx:
+            it = ((fs, f) for fs, files in _files.items() for f in files)
+            deque(callback.wrap(map_fn(get_file, it)), maxlen=0)
+        return result
+
     def get_file(self, rpath, lpath, **kwargs):
+        dvc_info = kwargs.pop("info", {}).pop("dvc_info", None)
         key = self._get_key_from_relative(rpath)
         fs_path = self._from_key(key)
+        dirpath = os.path.dirname(lpath)
+        if dirpath:
+            # makedirs raises error if the string is empty
+            os.makedirs(dirpath, exist_ok=True)
+
         try:
             return self.repo.fs.get_file(fs_path, lpath, **kwargs)
         except FileNotFoundError:
@@ -476,7 +619,7 @@ class _DVCFileSystem(AbstractFileSystem):
                 raise
 
         dvc_path = _get_dvc_path(dvc_fs, subkey)
-        return dvc_fs.get_file(dvc_path, lpath, **kwargs)
+        return dvc_fs.get_file(dvc_path, lpath, info=dvc_info, **kwargs)
 
     def du(self, path, total=True, maxdepth=None, withdirs=False, **kwargs):
         if maxdepth is not None:
@@ -525,16 +668,64 @@ class DVCFileSystem(FileSystem):
     protocol = "local"
     PARAM_CHECKSUM = "md5"
 
-    def _prepare_credentials(self, **config) -> Dict[str, Any]:
+    def _prepare_credentials(self, **config) -> dict[str, Any]:
         return config
 
     @functools.cached_property
     def fs(self) -> "_DVCFileSystem":
         return _DVCFileSystem(**self.fs_args)
 
+    @property
+    def immutable(self):
+        from dvc.scm import NoSCM
+
+        if isinstance(self.fs.repo.scm, NoSCM):
+            return False
+
+        return self.fs._repo_kwargs.get("rev") == self.fs.repo.get_rev()
+
     def getcwd(self):
         return self.fs.getcwd()
 
+    def _get(
+        self,
+        from_info: Union[AnyFSPath, list[AnyFSPath]],
+        to_info: Union[AnyFSPath, list[AnyFSPath]],
+        callback: "Callback" = DEFAULT_CALLBACK,
+        recursive: bool = False,
+        batch_size: Optional[int] = None,
+        **kwargs,
+    ) -> list[tuple[str, str, Optional[dict]]]:
+        # FileSystem.get is non-recursive by default if arguments are lists
+        # otherwise, it's recursive.
+        recursive = not (isinstance(from_info, list) and isinstance(to_info, list))
+        return self.fs._get(
+            from_info,
+            to_info,
+            callback=callback,
+            recursive=recursive,
+            batch_size=batch_size,
+            **kwargs,
+        )
+
+    def get(
+        self,
+        from_info: Union[AnyFSPath, list[AnyFSPath]],
+        to_info: Union[AnyFSPath, list[AnyFSPath]],
+        callback: "Callback" = DEFAULT_CALLBACK,
+        recursive: bool = False,
+        batch_size: Optional[int] = None,
+        **kwargs,
+    ) -> None:
+        self._get(
+            from_info,
+            to_info,
+            callback=callback,
+            batch_size=batch_size,
+            recursive=recursive,
+            **kwargs,
+        )
+
     @property
     def fsid(self) -> str:
         return self.fs.fsid
@@ -551,9 +742,10 @@ class DVCFileSystem(FileSystem):
         return self.fs.repo_url
 
     def from_os_path(self, path: str) -> str:
-        if os.path.isabs(path):
+        if os.path.isabs(path) or (
+            os.name == "nt" and posixpath.isabs(path) and ntpath.sep not in path
+        ):
             path = os.path.relpath(path, self.repo.root_dir)
-
         return as_posix(path)
 
     def close(self):

dvc/fs/dvc_path.py

@@ -0,0 +1,54 @@
+"""UPath implementation for DVCFileSystem.
+
+This provides a `pathlib.Path` like interface to
+work with DVCFileSystem.
+
+Examples
+--------
+
+>>> from upath import UPath
+
+>>> local = UPath("dvc://path/to/local/repo")
+>>> https = UPath("dvc+https://github.com/iterative/example-get-started", rev="main")
+>>> ssh = UPath("dvc+ssh://[email protected]:iterative/example-get-started.git")
+"""
+
+from urllib.parse import urlsplit
+
+from upath import UPath  # ty: ignore[unresolved-import]
+
+
+class DVCPath(UPath):
+    @classmethod
+    def _transform_init_args(cls, args, protocol, storage_options):
+        if not args:
+            args = ("/",)
+        elif (
+            args
+            and "url" not in storage_options
+            and protocol in {"dvc+http", "dvc+https", "dvc+ssh"}
+        ):
+            url, *rest = args
+            url = urlsplit(str(url))
+            proto = protocol.split("+")[1]
+            if proto == "ssh":
+                base_url = url.netloc + url.path
+            else:
+                base_url = url._replace(scheme=proto).geturl()
+            storage_options["url"] = base_url
+            # Assume the given path is a root url
+            args = ("/", *rest)
+        return super()._transform_init_args(args, "dvc", storage_options)
+
+    def __str__(self):
+        s = super().__str__()
+        if url := self.storage_options.get("url"):
+            return s.replace("dvc://", f"dvc+{url}", 1)
+        return s
+
+    def with_segments(self, *pathsegments):
+        obj = super().with_segments(*pathsegments)
+        # cache filesystem, as dvcfs does not cache filesystem
+        # caveat: any joinpath operation will instantiate filesystem
+        obj._fs_cached = self.fs
+        return obj

dvc/fs/git.py

@@ -39,9 +39,7 @@ class GitFileSystem(FileSystem):
         )
 
     @functools.cached_property
-    def fs(
-        self,
-    ) -> "FsspecGitFileSystem":
+    def fs(self) -> "FsspecGitFileSystem":
         from scmrepo.fs import GitFileSystem as FsspecGitFileSystem
 
         return FsspecGitFileSystem(**self.fs_args)

dvc/ignore.py

@@ -1,10 +1,19 @@
+import functools
 import os
 import re
-from collections import namedtuple
+from collections.abc import Iterable, Iterator
 from itertools import chain, groupby, takewhile
-from typing import TYPE_CHECKING, List, Optional
+from typing import TYPE_CHECKING, Any, Literal, NamedTuple, Optional, Union, overload
+
+try:
+    from pathspec.patterns.gitignore.spec import (  # type: ignore[import-not-found]
+        GitIgnoreSpecPattern,
+    )
+except ImportError:  # pathspec<1
+    from pathspec.patterns import (
+        GitWildMatchPattern as GitIgnoreSpecPattern,
+    )
 
-from pathspec.patterns import GitWildMatchPattern
 from pathspec.util import normalize_file
 from pygtrie import Trie
 
@@ -13,7 +22,9 @@ from dvc.log import logger
 from dvc.pathspec_math import PatternInfo, merge_patterns
 
 if TYPE_CHECKING:
-    from dvc.fs import AnyFSPath, FileSystem
+    from typing_extensions import Self
+
+    from dvc.fs import FileSystem
 
 logger = logger.getChild(__name__)
 
@@ -21,36 +32,86 @@ logger = logger.getChild(__name__)
 class DvcIgnore:
     DVCIGNORE_FILE = ".dvcignore"
 
-    def __call__(self, root, dirs, files):
+    def __call__(
+        self, root: str, dirs: list[str], files: list[str]
+    ) -> tuple[list[str], list[str]]:
         raise NotImplementedError
 
 
 class DvcIgnorePatterns(DvcIgnore):
-    def __init__(self, pattern_list, dirname, sep):
-        from pathspec.patterns.gitwildmatch import _DIR_MARK
+    def __init__(
+        self, pattern_list: Iterable[Union[PatternInfo, str]], dirname: str, sep: str
+    ) -> None:
+        try:
+            from pathspec.patterns.gitignore.spec import (  # type: ignore[import-not-found]
+                _DIR_MARK,
+            )
+        except ImportError:  # pathspec<1
+            from pathspec.patterns.gitwildmatch import (  # type: ignore[attr-defined, no-redef]
+                _DIR_MARK,
+            )
 
-        if pattern_list and isinstance(pattern_list[0], str):
-            pattern_list = [PatternInfo(pattern, "") for pattern in pattern_list]
-
-        self.sep = sep
-        self.pattern_list = pattern_list
-        self.dirname = dirname
-
-        self.regex_pattern_list = []
-        for count, pattern in enumerate(pattern_list):
-            pattern, group = GitWildMatchPattern.pattern_to_regex(pattern.patterns)
-            if pattern:
-                pattern = pattern.replace(f"<{_DIR_MARK}>", f"<{_DIR_MARK}{count}>")
-                self.regex_pattern_list.append((pattern, group))
-
-        self.ignore_spec = [
-            (ignore, re.compile("|".join(item[0] for item in group)))
-            for ignore, group in groupby(self.regex_pattern_list, lambda x: x[1])
-            if ignore is not None
+        pattern_infos = [
+            pattern if isinstance(pattern, PatternInfo) else PatternInfo(pattern, "")
+            for pattern in pattern_list
         ]
 
+        self.sep = sep
+        self.pattern_list: list[PatternInfo] = []
+        self.dirname = dirname
+        self.find_matching_pattern = functools.cache(self._find_matching_pattern)
+
+        regex_pattern_list: list[tuple[str, bool, bool, PatternInfo]] = []
+        for count, pattern_info in enumerate(pattern_infos):
+            regex, ignore = GitIgnoreSpecPattern.pattern_to_regex(pattern_info.patterns)
+            if regex is not None and ignore is not None:
+                self.pattern_list.append(pattern_info)
+                regex = regex.replace(f"<{_DIR_MARK}>", f"<{_DIR_MARK}{count}>")
+                regex_pattern_list.append(
+                    (regex, ignore, pattern_info.patterns.endswith("/"), pattern_info)
+                )
+
+        def keyfunc(item: tuple[str, bool, bool, PatternInfo]) -> tuple[bool, bool]:
+            _, ignore, dir_only_pattern, _ = item
+            return ignore, dir_only_pattern
+
+        self.ignore_spec: list[
+            tuple[
+                re.Pattern[str],
+                bool,
+                bool,
+                dict[Optional[str], tuple[str, PatternInfo]],
+            ]
+        ]
+        self.ignore_spec = []
+        for (ignore, dir_only_pattern), group in groupby(
+            regex_pattern_list, key=keyfunc
+        ):
+            if ignore:
+                # For performance, we combine all exclude patterns.
+                # But we still need to figure out which pattern matched which rule,
+                # (eg: to show in `dvc check-ignore`).
+                # So, we use named groups and keep a map of group name to pattern.
+                pattern_map: dict[Optional[str], tuple[str, PatternInfo]] = {
+                    f"rule_{i}": (regex, pi)
+                    for i, (regex, _, _, pi) in enumerate(group)
+                }
+                combined_regex = "|".join(
+                    f"(?P<{name}>{regex})" for name, (regex, _) in pattern_map.items()
+                )
+                self.ignore_spec.append(
+                    (re.compile(combined_regex), ignore, dir_only_pattern, pattern_map)
+                )
+            else:
+                # unignored patterns are not combined with `|`.
+                for regex, _, _, pi in group:
+                    pattern_map = {None: (regex, pi)}
+                    self.ignore_spec.append(
+                        (re.compile(regex), ignore, dir_only_pattern, pattern_map)
+                    )
+
     @classmethod
-    def from_file(cls, path, fs, name):
+    def from_file(cls, path: str, fs: "FileSystem", name: str) -> "Self":
         assert fs.isabs(path)
         dirname = fs.normpath(fs.dirname(path))
         with fs.open(path, encoding="utf-8") as fobj:
@@ -62,13 +123,15 @@ class DvcIgnorePatterns(DvcIgnore):
 
         return cls(path_spec_lines, dirname, fs.sep)
 
-    def __call__(self, root: List[str], dirs: List[str], files: List[str]):
+    def __call__(
+        self, root: str, dirs: list[str], files: list[str]
+    ) -> tuple[list[str], list[str]]:
         files = [f for f in files if not self.matches(root, f)]
         dirs = [d for d in dirs if not self.matches(root, d, True)]
 
         return dirs, files
 
-    def _get_normalize_path(self, dirname, basename):
+    def _get_normalize_path(self, dirname: str, basename: str) -> Optional[str]:
         # NOTE: `relpath` is too slow, so we have to assume that both
         # `dirname` and `self.dirname` are relative or absolute together.
 
@@ -81,62 +144,105 @@ class DvcIgnorePatterns(DvcIgnore):
             # NOTE: `os.path.join` is ~x5.5 slower
             path = f"{rel}{self.sep}{basename}"
         else:
-            return False
+            return None
 
         if os.name == "nt":
             return normalize_file(path)
         return path
 
-    def matches(self, dirname, basename, is_dir=False, details: bool = False):
+    @overload
+    def matches(
+        self,
+        dirname: str,
+        basename: str,
+        is_dir: bool = False,
+        details: Literal[False] = ...,
+    ) -> bool: ...
+
+    @overload
+    def matches(
+        self,
+        dirname: str,
+        basename: str,
+        is_dir: bool = False,
+        details: Literal[True] = ...,
+    ) -> tuple[bool, list[PatternInfo]]: ...
+
+    @overload
+    def matches(
+        self,
+        dirname: str,
+        basename: str,
+        is_dir: bool = False,
+        details: bool = False,
+    ) -> Union[bool, tuple[bool, list[PatternInfo]]]: ...
+
+    def matches(
+        self,
+        dirname: str,
+        basename: str,
+        is_dir: bool = False,
+        details: bool = False,
+    ) -> Union[bool, tuple[bool, list[PatternInfo]]]:
         path = self._get_normalize_path(dirname, basename)
-        if not path:
-            return False
-
-        if details:
-            return self._ignore_details(path, is_dir)
-        return self.ignore(path, is_dir)
-
-    def ignore(self, path, is_dir):
-        def matches(pattern, path, is_dir) -> bool:
-            matches_ = bool(pattern.match(path))
-
-            if is_dir:
-                matches_ |= bool(pattern.match(f"{path}/"))
-
-            return matches_
-
         result = False
+        _match: list[PatternInfo] = []
+        if path:
+            result, _match = self._ignore(path, is_dir)
+        return (result, _match) if details else result
 
-        for ignore, pattern in self.ignore_spec[::-1]:
-            if matches(pattern, path, is_dir):
-                result = ignore
-                break
-        return result
+    def _find_matching_pattern(
+        self, path: str, is_dir: bool
+    ) -> tuple[bool, list[PatternInfo]]:
+        paths = [path]
+        if is_dir and not path.endswith("/"):
+            paths.append(f"{path}/")
 
-    def _ignore_details(self, path, is_dir: bool):
-        result = []
-        for (regex, _), pattern_info in list(
-            zip(self.regex_pattern_list, self.pattern_list)
+        for pattern, ignore, dir_only_pattern, pattern_map in reversed(
+            self.ignore_spec
         ):
-            # skip system pattern
-            if not pattern_info.file_info:
+            if dir_only_pattern and not is_dir:
                 continue
+            for p in paths:
+                match = pattern.match(p)
+                if not match:
+                    continue
+                if ignore:
+                    group_name, _match = next(
+                        (
+                            (name, _match)
+                            for name, _match in match.groupdict().items()
+                            if name.startswith("rule_") and _match is not None
+                        )
+                    )
+                else:
+                    # unignored patterns are not combined with `|`,
+                    # so there are no groups.
+                    group_name = None
+                _regex, pattern_info = pattern_map[group_name]
+                return ignore, [pattern_info]
+        return False, []
 
-            regex = re.compile(regex)
+    def _ignore(self, path: str, is_dir: bool) -> tuple[bool, list[PatternInfo]]:
+        parts = path.split("/")
+        result = False
+        matches: list[PatternInfo] = []
+        for i in range(1, len(parts) + 1):
+            rel_path = "/".join(parts[:i])
+            result, _matches = self.find_matching_pattern(
+                rel_path, is_dir or i < len(parts)
+            )
+            if i < len(parts) and not result:
+                continue
+            matches.extend(_matches)
+            if result:
+                break
+        return result, matches
 
-            matches = bool(regex.match(path))
-            if is_dir:
-                matches |= bool(regex.match(f"{path}/"))
-
-            if matches:
-                result.append(pattern_info.file_info)
-
-        return result
-
-    def __hash__(self):
+    def __hash__(self) -> int:
         return hash(self.dirname + ":" + str(self.pattern_list))
 
-    def __eq__(self, other):
+    def __eq__(self, other: object) -> bool:
         if not isinstance(other, DvcIgnorePatterns):
             return NotImplemented
         return (self.dirname == other.dirname) & (
@@ -144,21 +250,18 @@ class DvcIgnorePatterns(DvcIgnore):
             == [pattern.patterns for pattern in other.pattern_list]
         )
 
-    def __bool__(self):
+    def __bool__(self) -> bool:
         return bool(self.pattern_list)
 
 
-CheckIgnoreResult = namedtuple(  # noqa: PYI024
-    "CheckIgnoreResult", ["file", "match", "patterns"]
-)
-
-
-def _no_match(path):
-    return CheckIgnoreResult(path, False, ["::"])
+class CheckIgnoreResult(NamedTuple):
+    file: str
+    match: bool
+    pattern_infos: list[PatternInfo]
 
 
 class DvcIgnoreFilter:
-    def __init__(self, fs, root_dir):
+    def __init__(self, fs: "FileSystem", root_dir: str) -> None:
         from dvc.repo import Repo
 
         default_ignore_patterns = [
@@ -193,7 +296,7 @@ class DvcIgnoreFilter:
             ignore_subrepos=True,
         )
 
-    def _get_key(self, path):
+    def _get_key(self, path: str) -> tuple[str, ...]:
         parts = self.fs.relparts(path, self.root_dir)
         if parts == (os.curdir,):
             return ()
@@ -226,7 +329,7 @@ class DvcIgnoreFilter:
         self,
         dirname: str,
         ignore_trie: Trie,
-        dnames: Optional["List"],
+        dnames: Optional["list"],
         ignore_subrepos: bool,
     ) -> None:
         self._update_trie(dirname, ignore_trie)
@@ -241,7 +344,7 @@ class DvcIgnoreFilter:
             for dname in dnames:
                 self._update_sub_repo(self.fs.join(dirname, dname), ignore_trie)
 
-    def _update_sub_repo(self, path, ignore_trie: Trie):
+    def _update_sub_repo(self, path: str, ignore_trie: Trie) -> None:
         from dvc.repo import Repo
 
         if path == self.root_dir:
@@ -268,7 +371,9 @@ class DvcIgnoreFilter:
         else:
             ignore_trie[key] = new_pattern
 
-    def __call__(self, root, dirs, files, ignore_subrepos=True):
+    def __call__(
+        self, root: str, dirs: list[str], files: list[str], ignore_subrepos: bool = True
+    ) -> tuple[list[str], list[str]]:
         abs_root = self.fs.abspath(root)
         ignore_pattern = self._get_trie_pattern(
             abs_root, dnames=dirs, ignore_subrepos=ignore_subrepos
@@ -277,7 +382,24 @@ class DvcIgnoreFilter:
             dirs, files = ignore_pattern(abs_root, dirs, files)
         return dirs, files
 
-    def ls(self, fs, path, detail=True, **kwargs):
+    @overload
+    def ls(
+        self, fs: "FileSystem", path: str, detail: Literal[True], **kwargs: Any
+    ) -> list[dict[str, Any]]: ...
+
+    @overload
+    def ls(
+        self, fs: "FileSystem", path: str, detail: Literal[False], **kwargs
+    ) -> list[str]: ...
+
+    @overload
+    def ls(
+        self, fs: "FileSystem", path: str, detail: bool = True, **kwargs
+    ) -> Union[list[str], list[dict[str, Any]]]: ...
+
+    def ls(
+        self, fs: "FileSystem", path: str, detail: bool = True, **kwargs: Any
+    ) -> Union[list[str], list[dict[str, Any]]]:
         fs_dict = {}
         dirs = []
         nondirs = []
@@ -297,22 +419,29 @@ class DvcIgnoreFilter:
 
         return [fs_dict[name] for name in chain(dirs, nondirs)]
 
-    def walk(self, fs: "FileSystem", path: "AnyFSPath", **kwargs):
+    def walk(
+        self, fs: "FileSystem", path: str, **kwargs: Any
+    ) -> Iterator[
+        Union[
+            tuple[str, list[str], list[str]],
+            tuple[str, dict[str, dict], dict[str, dict]],
+        ]
+    ]:
         detail = kwargs.get("detail", False)
         ignore_subrepos = kwargs.pop("ignore_subrepos", True)
         if fs.protocol == Schemes.LOCAL:
             for root, dirs, files in fs.walk(path, **kwargs):
                 if detail:
-                    all_dnames = set(dirs.keys())
-                    all_fnames = set(files.keys())
+                    assert isinstance(dirs, dict)
+                    assert isinstance(files, dict)
                     dnames, fnames = self(
                         root,
-                        all_dnames,
-                        all_fnames,
+                        list(dirs),
+                        list(files),
                         ignore_subrepos=ignore_subrepos,
                     )
-                    list(map(dirs.pop, all_dnames - set(dnames)))
-                    list(map(files.pop, all_fnames - set(fnames)))
+                    list(map(dirs.pop, dirs.keys() - set(dnames)))
+                    list(map(files.pop, files.keys() - set(fnames)))
                 else:
                     dirs[:], files[:] = self(
                         root, dirs, files, ignore_subrepos=ignore_subrepos
@@ -321,7 +450,7 @@ class DvcIgnoreFilter:
         else:
             yield from fs.walk(path, **kwargs)
 
-    def find(self, fs: "FileSystem", path: "AnyFSPath", **kwargs):
+    def find(self, fs: "FileSystem", path: str, **kwargs: Any) -> Iterator[str]:
         if fs.protocol == Schemes.LOCAL:
             for root, _, files in self.walk(fs, path, **kwargs):
                 for file in files:
@@ -331,7 +460,7 @@ class DvcIgnoreFilter:
             yield from fs.find(path)
 
     def _get_trie_pattern(
-        self, dirname, dnames: Optional["List"] = None, ignore_subrepos=True
+        self, dirname: str, dnames: Optional[list[str]] = None, ignore_subrepos=True
     ) -> Optional["DvcIgnorePatterns"]:
         if ignore_subrepos:
             ignores_trie = self.ignores_trie_fs
@@ -367,7 +496,7 @@ class DvcIgnoreFilter:
 
     def _is_ignored(
         self, path: str, is_dir: bool = False, ignore_subrepos: bool = True
-    ):
+    ) -> bool:
         if self._outside_repo(path):
             return False
         dirname, basename = self.fs.split(self.fs.normpath(path))
@@ -389,30 +518,29 @@ class DvcIgnoreFilter:
         path = self.fs.abspath(path)
         return self._is_ignored(path, False, ignore_subrepos=ignore_subrepos)
 
-    def _outside_repo(self, path):
+    def _outside_repo(self, path: str) -> bool:
         return not self.fs.isin_or_eq(path, self.root_dir)
 
-    def check_ignore(self, target):
+    def check_ignore(self, target: str) -> CheckIgnoreResult:
         # NOTE: can only be used in `dvc check-ignore`, see
-        # https://github.com/iterative/dvc/issues/5046
+        # https://github.com/treeverse/dvc/issues/5046
         full_target = self.fs.abspath(target)
+        matched_patterns: list[PatternInfo] = []
+        ignore = False
         if not self._outside_repo(full_target):
             dirname, basename = self.fs.split(self.fs.normpath(full_target))
             pattern = self._get_trie_pattern(dirname)
             if pattern:
-                matches = pattern.matches(
-                    dirname, basename, self.fs.isdir(full_target), True
+                ignore, matched_patterns = pattern.matches(
+                    dirname, basename, self.fs.isdir(full_target), details=True
                 )
-
-                if matches:
-                    return CheckIgnoreResult(target, True, matches)
-        return _no_match(target)
+        return CheckIgnoreResult(target, ignore, matched_patterns)
 
     def is_ignored(
         self, fs: "FileSystem", path: str, ignore_subrepos: bool = True
     ) -> bool:
         # NOTE: can't use self.check_ignore(path).match for now, see
-        # https://github.com/iterative/dvc/issues/4555
+        # https://github.com/treeverse/dvc/issues/4555
         if fs.protocol != Schemes.LOCAL:
             return False
         if fs.isfile(path):
@@ -424,7 +552,7 @@ class DvcIgnoreFilter:
         )
 
 
-def init(path):
+def init(path: Union[str, os.PathLike[str]]) -> str:
     dvcignore = os.path.join(path, DvcIgnore.DVCIGNORE_FILE)
     if os.path.exists(dvcignore):
         return dvcignore
@@ -439,7 +567,7 @@ def init(path):
     return dvcignore
 
 
-def destroy(path):
+def destroy(path: Union[str, os.PathLike[str]]) -> None:
     from dvc.utils.fs import remove
 
     dvcignore = os.path.join(path, DvcIgnore.DVCIGNORE_FILE)

dvc/info.py

@@ -48,7 +48,10 @@ def get_dvc_info():
                 info.append("Cache types: " + error_link("no-dvc-cache"))
 
             info.append(f"Caches: {_get_caches(repo.cache)}")
-            info.append(f"Remotes: {_get_remotes(repo.config)}")
+
+            configured_remotes = _get_remotes(repo.config)
+            remotes = ", ".join(configured_remotes) if configured_remotes else None
+            info.append(f"Remotes: {remotes}")
 
             root_directory = repo.root_dir
             fs_root = _get_fs_type(os.path.abspath(root_directory))
@@ -75,12 +78,10 @@ def _get_caches(cache):
 
 
 def _get_remotes(config):
-    schemes = (
+    return [
         get_fs_cls(get_fs_config(config, name=remote)).protocol
         for remote in config["remote"]
-    )
-
-    return ", ".join(schemes) or "None"
+    ]
 
 
 def _get_linktype_support_info(repo):
@@ -123,7 +124,7 @@ def _get_supported_remotes():
         if not fs_cls.get_missing_deps():
             dependencies = []
             for requirement in fs_cls.REQUIRES:
-                dependencies.append(
+                dependencies.append(  # noqa: PERF401
                     f"{requirement} = {importlib_metadata.version(requirement)}"
                 )
 
@@ -157,7 +158,7 @@ def _get_fs_type(path):
             except PermissionError:
                 pass
 
-    # need to follow the symlink: https://github.com/iterative/dvc/issues/5065
+    # need to follow the symlink: https://github.com/treeverse/dvc/issues/5065
     path = pathlib.Path(path).resolve()
 
     for parent in itertools.chain([path], path.parents):

dvc/lock.py

@@ -2,12 +2,13 @@
 
 import hashlib
 import os
+import time
 from abc import ABC, abstractmethod
 from datetime import timedelta
+from typing import Optional, Union
 
 import flufl.lock
 import zc.lockfile
-from funcy import retry
 
 from dvc.exceptions import DvcException
 from dvc.progress import Tqdm
@@ -87,9 +88,10 @@ class Lock(LockBase):
     Uses zc.lockfile as backend.
     """
 
-    def __init__(self, lockfile, friendly=False, **kwargs):
+    def __init__(self, lockfile, friendly=False, wait=False, **kwargs):
         super().__init__(lockfile)
         self._friendly = friendly
+        self._wait = wait
         self._lock = None
         self._lock_failed = False
 
@@ -100,23 +102,38 @@ class Lock(LockBase):
     def _do_lock(self):
         try:
             self._lock_failed = False
-            with Tqdm(
-                bar_format="{desc}",
-                disable=not self._friendly,
-                desc="If DVC froze, see `hardlink_lock` in {}".format(
-                    format_link("https://man.dvc.org/config#core")
-                ),
-            ):
-                self._lock = zc.lockfile.LockFile(self._lockfile)
+            self._lock = zc.lockfile.LockFile(self._lockfile)
         except zc.lockfile.LockError:
             self._lock_failed = True
             raise LockError(FAILED_TO_LOCK_MESSAGE)  # noqa: B904
 
     def lock(self):
-        retries = 6
-        delay = DEFAULT_TIMEOUT / retries
-        lock_retry = retry(retries, LockError, timeout=delay)(self._do_lock)
-        lock_retry()
+        """Acquire the lock, either waiting forever, or after default_retries."""
+        default_retries = 6
+        delay = DEFAULT_TIMEOUT / default_retries
+        attempts = 0
+
+        max_retries = float("inf") if self._wait else default_retries
+
+        with Tqdm(
+            bar_format="{desc}",
+            disable=not self._friendly,
+            desc="Waiting to acquire lock. "
+            "If DVC froze, see `hardlink_lock` in {}.".format(
+                format_link("https://man.dvc.org/config#core")
+            ),
+        ) as pbar:
+            while True:
+                try:
+                    self._do_lock()
+                    return
+                except LockError:
+                    attempts += 1
+                    if attempts > max_retries:
+                        raise
+                    time.sleep(delay)
+                finally:
+                    pbar.update()
 
     def unlock(self):
         if self._lock_failed:
@@ -149,7 +166,7 @@ class HardlinkLock(flufl.lock.Lock, LockBase):
         tmp_dir (str): a directory to store claim files.
     """
 
-    def __init__(self, lockfile, tmp_dir=None, **kwargs):
+    def __init__(self, lockfile, tmp_dir=None, wait=False, **kwargs):
         import socket
 
         self._tmp_dir = tmp_dir
@@ -165,18 +182,28 @@ class HardlinkLock(flufl.lock.Lock, LockBase):
         # number to generate the resulting lock file name, which is unique
         # enough for our application.
         #
-        # [1] https://github.com/iterative/dvc/issues/2582
+        # [1] https://github.com/treeverse/dvc/issues/2582
         self._hostname = socket.gethostname()
 
         self._lifetime = timedelta(days=365)  # Lock for good by default
         self._separator = flufl.lock.SEP
         self._set_claimfile()
         self._owned = True
+        self._wait = wait
         self._retry_errnos = []
+        self._friendly = kwargs.get("friendly", False)
 
-    def lock(self):
+    def lock(self, timeout: Optional[Union[timedelta, int]] = None):
         try:
-            super().lock(timedelta(seconds=DEFAULT_TIMEOUT))
+            if not self._wait:
+                timeout = timeout or timedelta(seconds=DEFAULT_TIMEOUT)
+
+            with Tqdm(
+                bar_format="{desc}",
+                disable=not (self._wait and self._friendly),
+                desc="Waiting to acquire lock",
+            ):
+                super().lock(timeout)
         except flufl.lock.TimeOutError:
             raise LockError(FAILED_TO_LOCK_MESSAGE)  # noqa: B904
 
@@ -185,13 +212,11 @@ class HardlinkLock(flufl.lock.Lock, LockBase):
 
         if self._tmp_dir is not None:
             # Under Windows file path length is limited so we hash it
-            hasher = hashlib.md5(  # noqa: S324
-                self._claimfile.encode()
-            )
+            hasher = hashlib.md5(self._claimfile.encode(), usedforsecurity=False)
             filename = hasher.hexdigest()
             self._claimfile = os.path.join(self._tmp_dir, filename + ".lock")
 
 
-def make_lock(lockfile, tmp_dir=None, friendly=False, hardlink_lock=False):
+def make_lock(lockfile, tmp_dir=None, friendly=False, hardlink_lock=False, wait=False):
     cls = HardlinkLock if hardlink_lock else Lock
-    return cls(lockfile, tmp_dir=tmp_dir, friendly=friendly)
+    return cls(lockfile, tmp_dir=tmp_dir, friendly=friendly, wait=wait)

dvc/logger.py

@@ -1,10 +1,11 @@
 """Manages logging configuration for DVC repo."""
 
 import logging
-import logging.config
-import logging.handlers
 import os
 import sys
+from collections.abc import Iterator
+from contextlib import contextmanager
+from typing import ClassVar
 
 import colorama
 
@@ -35,7 +36,7 @@ def add_logging_level(level_name, level_num, method_name=None):
             self._log(level_num, message, args, **kwargs)
 
     def log_to_root(message, *args, **kwargs):
-        logging.log(level_num, message, *args, **kwargs)
+        logging.log(level_num, message, *args, **kwargs)  # noqa: LOG015
 
     # getLevelName resolves the numeric log level if already defined,
     # otherwise returns a string
@@ -52,7 +53,7 @@ def add_logging_level(level_name, level_num, method_name=None):
         setattr(logging, method_name, log_to_root)
 
 
-class LoggingException(Exception):
+class LoggingException(Exception):  # noqa: N818
     def __init__(self, record):
         msg = f"failed to log {record!s}"
         super().__init__(msg)
@@ -78,7 +79,7 @@ class ColorFormatter(logging.Formatter):
     """
 
     reset = colorama.Fore.RESET
-    color_codes = {
+    color_codes: ClassVar[dict[str, str]] = {
         "TRACE": colorama.Fore.GREEN,
         "DEBUG": colorama.Fore.BLUE,
         "WARNING": colorama.Fore.YELLOW,
@@ -91,7 +92,7 @@ class ColorFormatter(logging.Formatter):
         self.log_colors = log_colors
         self.show_traceback = show_traceback
 
-    def format(self, record) -> str:  # noqa: A003, C901
+    def format(self, record) -> str:  # noqa: C901
         record.message = record.getMessage()
         msg = self.formatMessage(record)
 
@@ -173,9 +174,19 @@ def _iter_causes(exc):
         exc = exc.__cause__
 
 
-def set_loggers_level(level: int = logging.INFO) -> None:
+@contextmanager
+def set_loggers_level(level: int = logging.INFO) -> Iterator[None]:
+    ret: dict[logging.Logger, int] = {}
     for name in ["dvc", "dvc_objects", "dvc_data"]:
-        logging.getLogger(name).setLevel(level)
+        _logger = logging.getLogger(name)
+        ret[_logger] = _logger.getEffectiveLevel()
+        _logger.setLevel(level)
+
+    try:
+        yield
+    finally:
+        for _logger, old_lvl in ret.items():
+            _logger.setLevel(old_lvl)
 
 
 def setup(level: int = logging.INFO, log_colors: bool = True) -> None: