Commit Graph
3347 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 4d5a0163ea diff: handle granular targets 2023-09-28 21:54:06 +03:00
David de la Iglesia Castro a6e0d68cd2 run-cache: Load newest instead of first found in os.listdir. (#9908)
Closes https://github.com/iterative/dvc/issues/9420
2023-09-26 15:07:13 +03:00
Peter Rowlands (변기호) 12b572596f repo: add API and CLI for reading artifacts (#9770)
* deps: add GTO as a dependency

* deps: bump dvc-studio-client to 0.13.0

* fs: return downloaded count in fs.download()

* utils: add type hints for resolve_output

* artifacts: add `dvc artifacts get`

* api: add `api.artifacts_show`

* artifacts get: support --remote, --remote-config

* gto: update to 1.3.0
2023-09-25 20:21:34 +05:45
dberenbaum e8362db9bc exp: force add include_untracked paths 2023-09-25 19:02:38 +09:00
BradyJ27anddaavoo 51dc78a135 exp: add rename command (#9812)
* Add renaming for finished experiments

* Update arg parser

* Add rename by rev

* Add rename tests

* Change remove to rename; Refactor arguments error

* Remove --rev; Use resolve_name; Add test

* Update testing; Fix python3.10 rename failure

* Update rename test

* Update tests and error handling

* exp rename: Add `--force`.

Fail if experiment new_name exists but allow to overwrite it.

---------

Co-authored-by: daavoo <[email protected]>
2023-09-18 17:01:32 +00:00
dberenbaum bf2af86397 drop irrelevant errors in tests/unit/render/test_match.py 2023-09-14 10:36:46 -04:00
dberenbaum 08d37912d6 add util function to group plot definitions 2023-09-14 10:36:46 -04:00
dberenbaum 0e66aa2b6a add integration test for dvclive 2.x->3.x plots 2023-09-14 10:36:46 -04:00
dberenbaum f9340ed7b9 add test cases for plots group_definitions 2023-09-14 10:36:46 -04:00
dberenbaum e1599ae280 add test for merging plot ids across config files 2023-09-14 10:36:46 -04:00
dberenbaum 7e7d0e1dd8 merge plot ids across dvc.yaml files 2023-09-14 10:36:46 -04:00
dberenbaum d81a1bbf56 make exp run --dry imply --temp 2023-09-13 15:06:46 +09:00
Ruslan Kuprieiev e9c7cd63a2 dvcfs: add tests for fsid 2023-09-11 21:20:12 +03:00
skshetry f2673acb53 metrics/params: support reading from cached files and directories, use repo-relative path consistently (#9909)
* metrics

* fix path handling

* fix dvc metrics diff

* params show/diff

* share common utils

* remove onerror_collect from exp_show

* extract read_params

* move around code

* fix path handling on Windows

* rename _show to _gather_metrics

* remove _collect_vars

* add tests

* Revert "remove _collect_vars"

This reverts commit 6d2cb500c32a882497406d66fb13e613790c3436.

* bring back _collect_vars

* fix tests

* fix paths
2023-09-11 17:41:37 +05:45
David de la Iglesia Castro 7e52c233ae import: Expose --force flag (#9927) 2023-09-08 18:43:13 +05:45
David de la Iglesia Castro 7fb8a9a427 checkout: Expose --allow-missing arg. (#9919) 2023-09-07 09:55:50 +00:00
Ruslan Kuprieiev f47d21b0c2 [WIP] ls-url/get-url/import-url: introduce --fs-config (#9888)
ls-url/get-url/import-url: introduce --fs-config

Fixes #9886
2023-09-01 02:17:17 +03:00
Matt Seddon 5b69a16ea7 Add DVC_ROOT env var into queued experiment runs (#9879) 2023-08-28 22:37:03 +00:00
Dave Berenbaumandpre-commit-ci[bot] ef260eaf2e exp show: show metrics that include separator (#9819)
* exp show: show metrics that include separator

* sort using partition instead of lpartition

* handle ':' in file and metric names

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix sorting with leading :

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-08-24 10:31:56 -04:00
Dave Berenbaumandpre-commit-ci[bot] 428c094b8d update msg for multiple overlapping outputs (#9862)
* update msg for multiple overlapping outputs

* update msg for multiple overlapping outputs

* fix type checking

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use sets of stages to fix mypy errors

* fix tests/func/test_add.py::test_try_adding_multiple_overlaps

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-08-24 00:06:32 +03:00
pre-commit-ci[bot]andSaugat Pachhai 3783d65dde [pre-commit.ci] pre-commit autoupdate (#9841)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.0.282 → v0.0.285](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.282...v0.0.285)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix ruff issues

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-08-22 15:22:57 +05:45
skshetry fc9af7a976 assert that the test is getting parametrized correctly (#9859)
Otherwise this may end up generating 0 or not enough commands, and passing tests
2023-08-19 13:57:48 +05:45
skshetry fea59cfc52 tests: simplify test parametrization (#9858)
i.e. remove metafunc/pytest_generate_tests
2023-08-19 10:02:31 +05:45
Dave BerenbaumandRuslan Kuprieiev 1cbe64face Update tests/unit/command/test_experiments.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2023-08-18 00:31:52 +03:00
dberenbaum 057b0112ea exp: make run/save less noisy 2023-08-18 00:31:52 +03:00
Ruslan Kuprieiev 1805edbd32 import: set core.remote when both --remote and --remote-config are specified
From https://github.com/iterative/dvc/pull/9853#issuecomment-1682797083
2023-08-18 00:16:33 +03:00
Ruslan Kuprieiev 1723abcd79 ls/ls-url: introduce --size
I really need it only for studio right now, but this is really nice to have in a daily life
as well.

E.g.

```
$ dvc ls-url . --size
108    Metadata-and-Results.dvc
97     AudioMP3.dvc
97     AudioWAV.dvc
12181  README.md
54     .gitignore
168    .gitattributes
99     VideoFlash.dvc
312    LICENSE.txt
139    .dvcignore
96     AudioMP3
96     Asset
192    .dvc
64     Metadata-and-Results
64     AudioWAV
192    Scripts
64     VideoFlash
384    .git
96     .vscode
```

```
$ dvc list . --size
139    .dvcignore
168    .gitattributes
54     .gitignore
96     .vscode
96     Asset
96     AudioMP3
97     AudioMP3.dvc
64     AudioWAV
97     AudioWAV.dvc
312    LICENSE.txt
64     Metadata-and-Results
108    Metadata-and-Results.dvc
12181  README.md
192    Scripts
64     VideoFlash
99     VideoFlash.dvc
```
2023-08-18 00:08:48 +03:00
Ruslan Kuprieiev f0c1ff0219 fs: make _prepare_credentails/fs_args lazy (#9848)
* tests: unit: move s3 tests to dvc-s3

* deps: bump dvc-data to 2.14.0

* tests: unit: move azure tests to dvc-azure
2023-08-16 05:51:20 +03:00
Dave Berenbaum 79cc28501b don't unprotect outs if not cached (#9838)
* don't unprotect outs if not cached

* fix tests
2023-08-11 18:12:25 +03:00
Peter Rowlands a051b8b60f remove support for dvc machine 2023-08-11 14:42:52 +09:00
Tibor MachandTibor Mach 04e891cef9 exp: Unify dvc exp list and show output (#9808)
* use _describe in exp list

* fix pre-commit checks

* simplistic fix

* single _describe, too complex

* slightly simpler

* even simpler

* simple, but with regex

* less substitutions

* describe public

* name handling in UI (tests need updating)

* fixed old tests

* lazy imports

---------

Co-authored-by: Tibor Mach <[email protected]>
2023-08-10 19:23:27 +09:00
Peter Rowlands f5dda2d3cc deps: bump scmrepo minver to 1.2.1 2023-08-10 19:21:06 +09:00
Ruslan Kuprieiev 4f61a63afd config: resolve: handle ~ home dir
Fixes #9822
2023-08-10 03:28:32 +03:00
dberenbaum 5d2ca11557 exp: abstract random name gen from repo 2023-08-06 13:13:11 +02:00
009b238807 Feature/hydra config modules (#9783)
* hydra: enable config modules

Lets the user specify Hydra configs in installed Python modules as well
as inside directories. Adds `hydra.config_module` to `.dvc/config`.

Fixes #9740

old tests green

tests for config modules

* hydra: fix linter errors

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* hydra: fix mypy error

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* hydra: fix tests

* Update dvc/utils/hydra.py

Co-authored-by: David de la Iglesia Castro <[email protected]>

* address comments

* convince mypy not to worry

* split long line

* fix test

* check for raised error in test

---------

Co-authored-by: Dom Miketa <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David de la Iglesia Castro <[email protected]>
2023-08-06 10:39:09 +02:00
Ruslan Kuprieiev 43b371c43a deps: bump dvc-data to 2.11.0 2023-08-04 15:42:25 +03:00
skshetry d1160a28e0 dvc.yaml: add matrix-do (#9725)
* dvc.yaml: add matrix-do

* remove docs

* no do in matrix

* add types
2023-08-04 13:29:15 +05:45
skshetry bc86fa0643 completion: fix broken completion generation for zsh (#9797)
`dvc commit --help` was also completely broken, which is also fixed.

This adds unit tests for completion generation for both zsh/bash, and
tests that --help works for all commands.
2023-08-04 09:21:01 +05:45
Saugat Pachhai (सौगात) 4067a86926 fix flaky test for repro --keep-going and --ignore 2023-08-04 05:46:58 +03:00
Ruslan Kuprieiev b235487ecc ls/import/get: introduce --remote and --remote-config
Fixes #9656
2023-08-02 16:28:16 +03:00
pre-commit-ci[bot]andSaugat Pachhai 502b3df6c9 [pre-commit.ci] pre-commit autoupdate (#9756)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/asottile/pyupgrade: v3.9.0 → v3.10.1](https://github.com/asottile/pyupgrade/compare/v3.9.0...v3.10.1)
- [github.com/astral-sh/ruff-pre-commit: v0.0.278 → v0.0.281](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.278...v0.0.281)
- [github.com/PyCQA/flake8: 6.0.0 → 6.1.0](https://github.com/PyCQA/flake8/compare/6.0.0...6.1.0)

* fix issues reported by ruff

- also remove redundant flake8 plugins
- remove beautysh as we don't have any scripts now
- remove pyupgrade as it is redundant due to ruff

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-08-02 18:08:30 +05:45
Ruslan Kuprieiev 225814fdde dvc: don't use realpath where not needed 2023-07-30 02:24:00 +03:00
Ruslan Kuprieiev 4d011718a6 out: str: don't try to relpath paths outside of the repo
This is not very useful overall, but also results in a ValueError from
relpath on windows if your repo and output are on different drives.
2023-07-25 23:49:27 +03:00
Ruslan Kuprieiev 73dad4780f repo: brancher: reset parsed fields
Related #9754 and fixes `dvc fetch`.
2023-07-25 14:00:33 +03:00
Ruslan Kuprieiev 65ba9ba527 fs: get_cloud_fs: accept config instead of repo
Config is a more immutable thing and using `repo` is confusing and might be error
prone in the bigger picture.

Related to #9754
2023-07-25 00:58:47 +03:00
Peter Rowlands 7806afd4ad fs: prefer generic.copy over fsspec fs.get() 2023-07-24 17:01:34 +09:00
Ivan Shcheklein a9d0ac2ba9 fix(exceptions): add parent ctr init in InvalidArgumentError 2023-07-21 08:27:53 +02:00
Ruslan Kuprieiev 53ec0fa41d ls/import/get: introduce --config
Adds support for `--config` to all three commands. For `ls` and `get`, it is
a purely runtime option that will merge the config you provide with target
repo's configs (similar to how .dvc/config.local works). E.g.

```
$ cat myconfig
[remote "myremote"]
    access_key_id = 12345
    secret_access_key = 98765

$ dvc get https://github.com/efiop/mydataregistry recent-grads.csv --config myconfig
```

In case of `dvc import`, the semantics are the same, but the value is also
recorded in the resulting dvcfile. E.g.

```
$ dvc import https://github.com/efiop/mydataregistry recent-grads.csv --config myconfig
...
$ cat recent-grads.csv.dvc
md5: 85c86eeb3afa6d1f7d11beedc644d28e
frozen: true
deps:
- path: recent-grads.csv
  repo:
    url: https://github.com/efiop/mydataregistry
    config: myconfig
    rev_lock: af22e06bbfe34d62a140437abf32f48b535944a7
outs:
- md5: f447e23a170a9f25a7799f069a1ba934
  size: 26872
  hash: md5
  path: recent-grads.csv
```

This is the most general and powerful mechanism that we can give, that works
for everything from reconfiguring your default remote to completely changing
your remote layout and beyond (will be handy in the future).

Fixes #2466
2023-07-20 18:41:10 +03:00
Dave BerenbaumandRuslan Kuprieiev 41194b7975 dvcfs: handle NotADirectoryError from datafs.ls (#9746)
* dvc_fs: ignore NotADirectoryError in ls

* deps: bump dvc-data to >= 2.6.0

* add test

---------

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2023-07-19 20:54:55 +03:00
skshetry ea5f7d40c1 ruff: enable a few more rules, and merge tests' config with the main config (#9739) 2023-07-15 14:33:34 +05:45