Commit Graph
364 Commits
Author SHA1 Message Date
skshetry f2673acb53 metrics/params: support reading from cached files and directories, use repo-relative path consistently (#9909)
* metrics

* fix path handling

* fix dvc metrics diff

* params show/diff

* share common utils

* remove onerror_collect from exp_show

* extract read_params

* move around code

* fix path handling on Windows

* rename _show to _gather_metrics

* remove _collect_vars

* add tests

* Revert "remove _collect_vars"

This reverts commit 6d2cb500c32a882497406d66fb13e613790c3436.

* bring back _collect_vars

* fix tests

* fix paths
2023-09-11 17:41:37 +05:45
David de la Iglesia Castro 7e52c233ae import: Expose --force flag (#9927) 2023-09-08 18:43:13 +05:45
David de la Iglesia Castro 7fb8a9a427 checkout: Expose --allow-missing arg. (#9919) 2023-09-07 09:55:50 +00:00
Ruslan Kuprieiev f47d21b0c2 [WIP] ls-url/get-url/import-url: introduce --fs-config (#9888)
ls-url/get-url/import-url: introduce --fs-config

Fixes #9886
2023-09-01 02:17:17 +03:00
pre-commit-ci[bot]andSaugat Pachhai 3783d65dde [pre-commit.ci] pre-commit autoupdate (#9841)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.0.282 → v0.0.285](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.282...v0.0.285)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix ruff issues

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-08-22 15:22:57 +05:45
skshetry fc9af7a976 assert that the test is getting parametrized correctly (#9859)
Otherwise this may end up generating 0 or not enough commands, and passing tests
2023-08-19 13:57:48 +05:45
skshetry fea59cfc52 tests: simplify test parametrization (#9858)
i.e. remove metafunc/pytest_generate_tests
2023-08-19 10:02:31 +05:45
Dave BerenbaumandRuslan Kuprieiev 1cbe64face Update tests/unit/command/test_experiments.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2023-08-18 00:31:52 +03:00
dberenbaum 057b0112ea exp: make run/save less noisy 2023-08-18 00:31:52 +03:00
Peter Rowlands a051b8b60f remove support for dvc machine 2023-08-11 14:42:52 +09:00
Tibor MachandTibor Mach 04e891cef9 exp: Unify dvc exp list and show output (#9808)
* use _describe in exp list

* fix pre-commit checks

* simplistic fix

* single _describe, too complex

* slightly simpler

* even simpler

* simple, but with regex

* less substitutions

* describe public

* name handling in UI (tests need updating)

* fixed old tests

* lazy imports

---------

Co-authored-by: Tibor Mach <[email protected]>
2023-08-10 19:23:27 +09:00
skshetry bc86fa0643 completion: fix broken completion generation for zsh (#9797)
`dvc commit --help` was also completely broken, which is also fixed.

This adds unit tests for completion generation for both zsh/bash, and
tests that --help works for all commands.
2023-08-04 09:21:01 +05:45
Ruslan Kuprieiev b235487ecc ls/import/get: introduce --remote and --remote-config
Fixes #9656
2023-08-02 16:28:16 +03:00
Ruslan Kuprieiev 53ec0fa41d ls/import/get: introduce --config
Adds support for `--config` to all three commands. For `ls` and `get`, it is
a purely runtime option that will merge the config you provide with target
repo's configs (similar to how .dvc/config.local works). E.g.

```
$ cat myconfig
[remote "myremote"]
    access_key_id = 12345
    secret_access_key = 98765

$ dvc get https://github.com/efiop/mydataregistry recent-grads.csv --config myconfig
```

In case of `dvc import`, the semantics are the same, but the value is also
recorded in the resulting dvcfile. E.g.

```
$ dvc import https://github.com/efiop/mydataregistry recent-grads.csv --config myconfig
...
$ cat recent-grads.csv.dvc
md5: 85c86eeb3afa6d1f7d11beedc644d28e
frozen: true
deps:
- path: recent-grads.csv
  repo:
    url: https://github.com/efiop/mydataregistry
    config: myconfig
    rev_lock: af22e06bbfe34d62a140437abf32f48b535944a7
outs:
- md5: f447e23a170a9f25a7799f069a1ba934
  size: 26872
  hash: md5
  path: recent-grads.csv
```

This is the most general and powerful mechanism that we can give, that works
for everything from reconfiguring your default remote to completely changing
your remote layout and beyond (will be handy in the future).

Fixes #2466
2023-07-20 18:41:10 +03:00
Saugat Pachhai (सौगात) f1764bdc77 stage add: remove --external 2023-07-14 15:38:57 +03:00
skshetry eb212a7ba3 repro: introduce --keep-going/--ignore to continue on error (#9712)
* repro: introduce --keep-going/--ignore to continue on error

`-k`/`--keep-going` will skip all dependents of a stage if it fails.
It'll still try to reproduce the other dependencies of the pending
targets, before it exits with a non-zero status.

`--ignore-errors` will treat any error during stage execution as a
success, and will continue executing other dependencies as if it
succeeded. Exits with 0 even on failure.

Also fixes:
* --force-downstream used to force everything even if one upstream nodes changed.
  This used to work when we supported a single targets, but there could be independent
  pipelines, so  we should not force everything if any node changes.
  Now we'll force reproduce the stage only if one of its dependency changed upstream
  or it's upstream node was also forced.

* Makes `--downstream` more correct, in relation to `--allow-missing` and `--pull`.
  Now we use an active graph for downstream/upstream logic instead of a subgraph.
  Note that this does not affect the downstream logic though, as dvc always goes
  downstream, so subgraph was no different than active graph.
  The subgraph only did not have the upstream nodes.

* add tests

* add spacing in the message
2023-07-12 21:52:41 +05:45
Ruslan Kuprieiev 4de4f0fd65 fetch: introduce --type metrics/plots
Needed for https://github.com/iterative/studio/pull/6541 but is also generally useful.
2023-07-10 15:24:30 +03:00
Ruslan Kuprieiev 8661abf224 fetch: introduce --max-size (#9696)
Needed for iterative/studio#4782
2023-07-04 02:17:53 +03:00
Ruslan Kuprieiev cf976f17ae data: status: use ui.progress instead of ui.status (#9662)
With all fetch and checkout related things that we've done in the recent past,
we now have all the mechanisms necessary to provide actual progress (no total,
but still better than spinner) during `dvc data status` operations.
2023-06-26 00:10:44 +03:00
dberenbaum 04b50c5cd0 exp push/pull: default to head rev 2023-06-20 15:53:46 +09:00
dberenbaum bf2f08da09 exp list: fix ui for tags/branches 2023-06-20 15:52:18 +09:00
Ruslan Kuprieiev 1835c8c6e9 tests: unit: explicitly ask for dvc repo
Currently these rely on the root of the repo being in the dvc repo.
2023-06-19 03:44:10 +03:00
Ruslan Kuprieiev c827654fe7 tests: use mocker instead of unittest.mock where possible
Had the "pleasure" of finding a mocker.patch leak into other tests and one
of the suspicions is that mixing unittest.mock and mocker might be somehow
related.

In any case, we should use mocker where possible anyway.
2023-06-18 17:10:45 +03:00
Ruslan Kuprieiev af1fb722c8 dvc: drop cached external outputs
Fixes #9531

Docs https://github.com/iterative/dvc.org/pull/4574 Kudos @dberenbaum
2023-06-09 18:34:36 +03:00
Dave Berenbaum 528eab0418 import: drop --file (#9562) 2023-06-09 10:03:32 +05:45
skshetry 46f3157f43 add: rename jobs to remote_jobs to mirror CLI; move validation to CLI (#9550)
- Follow up on https://github.com/iterative/dvc/pull/9522, renames jobs kwarg to remote_jobs
  to mirror CLI.
- Also moves validation logic to CLI.
2023-06-06 19:30:39 +05:45
dberenbaum 2c4365513e add: deprecate --file 2023-06-06 14:57:07 +03:00
Dave Berenbaum 3a85326df3 add: rename jobs to remote-jobs (#9522) 2023-05-31 01:14:30 +03:00
skshetry 355ebc72eb add: remove --recursive, rename fname to file (#9513)
add: remove --recursive
2023-05-26 12:07:13 +05:45
skshetry 0e38525d8d cli: remove dvc run (#9508)
Note that `Repo.run()` API still exists, which will be removed in
successive PRs. A lot of tests depend on it that it'll take time
before removing it.
2023-05-25 16:27:14 +05:45
Dave Berenbaum 8acd943d52 exp list: show experiment shas (#9501)
* exp list: include sha; still needs remote support

* exp list: ignore rev for remotes

* exp list: update output format

* refactor
2023-05-25 15:05:12 +09:00
David de la Iglesia Castro 25313f5d82 exp: Drop checkpoints (#9271)
exp: Drop `checkpoints`.

Closes #9221
2023-05-23 11:32:18 +05:45
David de la Iglesia Castro f245567613 exp: Drop gc. (#9273)
Closes #7991
2023-05-22 16:08:25 +05:45
skshetry 1989c0fe98 stage add: introduce --run option (#9480)
* stage add: introduce --run option

Part of #5846.

* fix ui message
2023-05-22 12:27:20 +05:45
skshetry e089ee0365 remove data ls and support for --meta/--desc/--label/--type cli flags (#9481) 2023-05-19 20:10:34 +05:45
David de la Iglesia Castro cdba65f27e exp: Drop init. (#9269)
Drop `exp init`.

Closes #9267
2023-05-19 14:39:16 +05:45
skshetry 90aaf5a296 commands: remove --show-json/--show-md/--show-csv flags (#9475) 2023-05-18 16:36:14 +05:45
daavoo 64beb1578a repro: Add --allow-missing.
Point 2 in https://github.com/iterative/dvc/pull/9375#issuecomment-1531683194
2023-05-16 22:43:24 +02:00
dberenbaum f888544554 gc: disallow --remote without other flags 2023-05-16 17:18:57 +02:00
David de la Iglesia Castro aa517bc90e exp save: Support --message.
Custom commit message to use when committing the experiment.

Part of #8870
2023-05-08 20:08:47 +02:00
David de la Iglesia Castro c74dc20b97 exp run: Support --message.
Custom commit message to use when committing the experiment.

Part of #8870
2023-05-08 20:08:47 +02:00
David de la Iglesia Castro 8620b6e5af exp: Update --rev to support append action.
Closes #9390
2023-05-08 14:15:07 +02:00
dberenbaum 2db07b047e data status: exclude remote status by default 2023-05-06 18:21:11 +03:00
David de la Iglesia Castro c7e2ec9707 gc: Add --not-in-remote arg.
Closes #2036
2023-04-27 10:30:39 +02:00
Oded Messer d203bf9024 Fix: Prettifying unexpected error on FileExists for get/get-url/imp-url/add and introducing --force (#9365) 2023-04-27 00:41:09 +00:00
Peter Rowlands ec090c5680 exp: use new collection in show/cli commands
- Updates 'dvc exp show --json' to use new data format
2023-04-20 16:58:32 +09:00
dberenbaum 1c394986db fix tests 2023-04-15 23:46:47 +03:00
David de la Iglesia Castro 3312a90949 exp run: Add --copy-paths arg.
List of paths to copy inside the temp directory. Only used if `--temp` or `--queue` is specified.

Closes #5800
2023-04-05 23:17:07 +02:00
skshetry 086d51d9de plots json: return definition errors unrelated to renderers (#9293) 2023-03-31 16:34:26 +05:45
skshetry 43195a806d plots: return errors in json format (#9146)
* plots: return errors in json format

* plots --json: move plots data into data key

* rename  revision key to rev

* move imports to top-level

* add tests
2023-03-27 12:53:43 +05:45