8919 Commits
Author SHA1 Message Date
Peter Rowlands a051b8b60f remove support for dvc machine 3.15.0 2023-08-11 14:42:52 +09:00
Ruslan Kuprieiev bee04c6d5f deps: bump dvc-data to 2.13.1
Fixes #9733
2023-08-11 05:53:19 +03:00
Tibor MachandTibor Mach 04e891cef9 exp: Unify dvc exp list and show output (#9808)
* use _describe in exp list

* fix pre-commit checks

* simplistic fix

* single _describe, too complex

* slightly simpler

* even simpler

* simple, but with regex

* less substitutions

* describe public

* name handling in UI (tests need updating)

* fixed old tests

* lazy imports

---------

Co-authored-by: Tibor Mach <[email protected]>
2023-08-10 19:23:27 +09:00
Peter Rowlands f5dda2d3cc deps: bump scmrepo minver to 1.2.1 2023-08-10 19:21:06 +09:00
Ruslan Kuprieiev e9545c8b55 fetch: merge config instead of updating
Update will replace sections instead of merging them together, which
is not desirable here.

Related https://github.com/iterative/studio/pull/6541
3.14.0
2023-08-10 04:52:08 +03:00
Ruslan Kuprieiev 09cfd911bb config: use local config from local_dvc_dir
This used to work like so but by accident, when we didn't reload config
at all when collecting other revisions.

Regression from https://github.com/iterative/dvc/pull/9758
2023-08-10 04:36:42 +03:00
Ruslan Kuprieiev 4f61a63afd config: resolve: handle ~ home dir
Fixes #9822
2023-08-10 03:28:32 +03:00
Ruslan Kuprieiev c71b07b996 index: plot_keys: relparts -> parts
`_plot_sources` is messed up and needs a rewrite hard, but this will
do for now for studio purposes.

Related https://github.com/iterative/studio/pull/6541
3.13.3
2023-08-08 17:14:28 +03:00
Ruslan Kuprieiev f4c0946f65 repo: cache: support some view options 3.13.2 2023-08-08 01:09:50 +03:00
Ruslan Kuprieiev e35fb76dac repo: add basic cache_check_missing 3.13.1 2023-08-07 22:23:49 +03:00
Ruslan Kuprieiev 722839dda6 fetch: add basic onerror
Related https://github.com/iterative/studio/pull/6541
3.13.0
2023-08-07 03:24:39 +03:00
Ruslan Kuprieiev d5533348f7 deps: bump dvc-data to 2.12.0 2023-08-07 02:42:41 +03:00
dberenbaum 5d2ca11557 exp: abstract random name gen from repo 2023-08-06 13:13:11 +02:00
009b238807 Feature/hydra config modules (#9783)
* hydra: enable config modules

Lets the user specify Hydra configs in installed Python modules as well
as inside directories. Adds `hydra.config_module` to `.dvc/config`.

Fixes #9740

old tests green

tests for config modules

* hydra: fix linter errors

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* hydra: fix mypy error

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* hydra: fix tests

* Update dvc/utils/hydra.py

Co-authored-by: David de la Iglesia Castro <[email protected]>

* address comments

* convince mypy not to worry

* split long line

* fix test

* check for raised error in test

---------

Co-authored-by: Dom Miketa <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David de la Iglesia Castro <[email protected]>
2023-08-06 10:39:09 +02:00
Ruslan Kuprieiev 43b371c43a deps: bump dvc-data to 2.11.0 3.12.0 2023-08-04 15:42:25 +03:00
skshetry d1160a28e0 dvc.yaml: add matrix-do (#9725)
* dvc.yaml: add matrix-do

* remove docs

* no do in matrix

* add types
2023-08-04 13:29:15 +05:45
skshetry bc86fa0643 completion: fix broken completion generation for zsh (#9797)
`dvc commit --help` was also completely broken, which is also fixed.

This adds unit tests for completion generation for both zsh/bash, and
tests that --help works for all commands.
3.11.1
2023-08-04 09:21:01 +05:45
Saugat Pachhai (सौगात) 4067a86926 fix flaky test for repro --keep-going and --ignore 2023-08-04 05:46:58 +03:00
Ruslan Kuprieiev 9d1d51caa7 deps: bump dvc-data to 2.10.0 2023-08-04 03:40:36 +03:00
Peter Rowlands 761422638f output: add progress for commit 3.11.0 2023-08-03 18:19:19 +09:00
Peter Rowlands d01e2355d4 push/pull: generate explicit progress bars for dvc-data transfers 2023-08-03 18:19:19 +09:00
Peter Rowlands aa94bfd66b deps: bump dvc-data to 2.9.1 2023-08-03 18:19:19 +09:00
skshetry b9b9f91084 ruff: enable autofix (#9790) 2023-08-03 11:08:29 +05:45
Ruslan Kuprieiev 8a382f51fd deps: bump dvc-data to 2.9.0 2023-08-03 02:05:14 +03:00
Ruslan Kuprieiev b235487ecc ls/import/get: introduce --remote and --remote-config
Fixes #9656
2023-08-02 16:28:16 +03:00
pre-commit-ci[bot]andSaugat Pachhai 502b3df6c9 [pre-commit.ci] pre-commit autoupdate (#9756)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/asottile/pyupgrade: v3.9.0 → v3.10.1](https://github.com/asottile/pyupgrade/compare/v3.9.0...v3.10.1)
- [github.com/astral-sh/ruff-pre-commit: v0.0.278 → v0.0.281](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.278...v0.0.281)
- [github.com/PyCQA/flake8: 6.0.0 → 6.1.0](https://github.com/PyCQA/flake8/compare/6.0.0...6.1.0)

* fix issues reported by ruff

- also remove redundant flake8 plugins
- remove beautysh as we don't have any scripts now
- remove pyupgrade as it is redundant due to ruff

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-08-02 18:08:30 +05:45
dependabot[bot] 5c2fd67e6d build(deps-dev): bump pylint from 2.17.4 to 2.17.5 (#9769)
Bumps [pylint](https://github.com/pylint-dev/pylint) from 2.17.4 to 2.17.5.
- [Release notes](https://github.com/pylint-dev/pylint/releases)
- [Commits](https://github.com/pylint-dev/pylint/compare/v2.17.4...v2.17.5)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-31 16:35:40 +05:45
Fabian Zills ed34c0f0bd dvc commit update description (#9760)
* Update commit.py

* fix formatting
2023-07-31 08:45:52 +00:00
Ruslan Kuprieiev 7c60dfd3c5 deps: bump dvc-data to >=2.8.1 3.10.1 2023-07-30 04:18:09 +03:00
Ruslan Kuprieiev 225814fdde dvc: don't use realpath where not needed 3.10.0 2023-07-30 02:24:00 +03:00
Ruslan Kuprieiev ca0ad239f7 deps: bump dvc-data to 0.28.0 3.9.1 2023-07-28 01:37:46 +03:00
Ruslan Kuprieiev f57ac609e9 deps: bump dvc-data to 0.27.1 3.9.0 2023-07-27 20:44:34 +03:00
Ruslan Kuprieiev cc4f505f99 api: open/read: introduce remote_config 2023-07-26 16:29:14 +03:00
Ruslan Kuprieiev 4d011718a6 out: str: don't try to relpath paths outside of the repo
This is not very useful overall, but also results in a ValueError from
relpath on windows if your repo and output are on different drives.
3.8.1
2023-07-25 23:49:27 +03:00
Ruslan Kuprieiev 9c89841637 fetch: support passing config
For now not exposing it in CLI since there is no need/use yet.

Requirement for https://github.com/iterative/studio/pull/6541
3.8.0
2023-07-25 20:03:38 +03:00
Peter Rowlands 92cae74512 exp.collect: strip metrics from queued experiments 2023-07-25 20:57:15 +09:00
Ruslan Kuprieiev 73dad4780f repo: brancher: reset parsed fields
Related #9754 and fixes `dvc fetch`.
2023-07-25 14:00:33 +03:00
Ruslan Kuprieiev 65ba9ba527 fs: get_cloud_fs: accept config instead of repo
Config is a more immutable thing and using `repo` is confusing and might be error
prone in the bigger picture.

Related to #9754
2023-07-25 00:58:47 +03:00
Peter Rowlands 7806afd4ad fs: prefer generic.copy over fsspec fs.get() 2023-07-24 17:01:34 +09:00
Ivan Shcheklein a9d0ac2ba9 fix(exceptions): add parent ctr init in InvalidArgumentError 3.7.0 2023-07-21 08:27:53 +02:00
Ruslan Kuprieiev 41df96fcdb cli: ls/import/get: fix help
Leftover from https://github.com/iterative/dvc/pull/9747
2023-07-20 21:53:14 +03:00
Ruslan Kuprieiev 53ec0fa41d ls/import/get: introduce --config
Adds support for `--config` to all three commands. For `ls` and `get`, it is
a purely runtime option that will merge the config you provide with target
repo's configs (similar to how .dvc/config.local works). E.g.

```
$ cat myconfig
[remote "myremote"]
    access_key_id = 12345
    secret_access_key = 98765

$ dvc get https://github.com/efiop/mydataregistry recent-grads.csv --config myconfig
```

In case of `dvc import`, the semantics are the same, but the value is also
recorded in the resulting dvcfile. E.g.

```
$ dvc import https://github.com/efiop/mydataregistry recent-grads.csv --config myconfig
...
$ cat recent-grads.csv.dvc
md5: 85c86eeb3afa6d1f7d11beedc644d28e
frozen: true
deps:
- path: recent-grads.csv
  repo:
    url: https://github.com/efiop/mydataregistry
    config: myconfig
    rev_lock: af22e06bbfe34d62a140437abf32f48b535944a7
outs:
- md5: f447e23a170a9f25a7799f069a1ba934
  size: 26872
  hash: md5
  path: recent-grads.csv
```

This is the most general and powerful mechanism that we can give, that works
for everything from reconfiguring your default remote to completely changing
your remote layout and beyond (will be handy in the future).

Fixes #2466
2023-07-20 18:41:10 +03:00
Dave BerenbaumandRuslan Kuprieiev 41194b7975 dvcfs: handle NotADirectoryError from datafs.ls (#9746)
* dvc_fs: ignore NotADirectoryError in ls

* deps: bump dvc-data to >= 2.6.0

* add test

---------

Co-authored-by: Ruslan Kuprieiev <[email protected]>
3.6.0
2023-07-19 20:54:55 +03:00
skshetry ea5f7d40c1 ruff: enable a few more rules, and merge tests' config with the main config (#9739) 2023-07-15 14:33:34 +05:45
dependabot[bot] eb06351062 build(deps-dev): bump mypy from 1.3.0 to 1.4.1 (#9664)
Bumps [mypy](https://github.com/python/mypy) from 1.3.0 to 1.4.1.
- [Commits](https://github.com/python/mypy/compare/v1.3.0...v1.4.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-14 20:30:47 +05:45
pre-commit-ci[bot]andSaugat Pachhai ca3704997b [pre-commit.ci] pre-commit autoupdate (#9632)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/asottile/pyupgrade: v3.3.1 → v3.9.0](https://github.com/asottile/pyupgrade/compare/v3.3.1...v3.9.0)
- https://github.com/charliermarsh/ruff-pre-commit → https://github.com/astral-sh/ruff-pre-commit
- [github.com/astral-sh/ruff-pre-commit: v0.0.261 → v0.0.277](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.261...v0.0.277)
- [github.com/codespell-project/codespell: v2.2.4 → v2.2.5](https://github.com/codespell-project/codespell/compare/v2.2.4...v2.2.5)

* fix ruff complaints

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-07-14 20:11:15 +05:45
Saugat Pachhai (सौगात) f1764bdc77 stage add: remove --external 2023-07-14 15:38:57 +03:00
skshetry eb212a7ba3 repro: introduce --keep-going/--ignore to continue on error (#9712)
* repro: introduce --keep-going/--ignore to continue on error

`-k`/`--keep-going` will skip all dependents of a stage if it fails.
It'll still try to reproduce the other dependencies of the pending
targets, before it exits with a non-zero status.

`--ignore-errors` will treat any error during stage execution as a
success, and will continue executing other dependencies as if it
succeeded. Exits with 0 even on failure.

Also fixes:
* --force-downstream used to force everything even if one upstream nodes changed.
  This used to work when we supported a single targets, but there could be independent
  pipelines, so  we should not force everything if any node changes.
  Now we'll force reproduce the stage only if one of its dependency changed upstream
  or it's upstream node was also forced.

* Makes `--downstream` more correct, in relation to `--allow-missing` and `--pull`.
  Now we use an active graph for downstream/upstream logic instead of a subgraph.
  Note that this does not affect the downstream logic though, as dvc always goes
  downstream, so subgraph was no different than active graph.
  The subgraph only did not have the upstream nodes.

* add tests

* add spacing in the message
2023-07-12 21:52:41 +05:45
skshetry 4c8f539c56 plots: log failures during datapoint conversions (#9726)
Fixes #9441.

Note that we do merge props, and use that for all revision. So you may
see warning for multiple revisions.
2023-07-12 19:14:31 +05:45
skshetry 45aefc8423 parsing: pass config instead of loading every time (#9729) 2023-07-12 17:22:42 +05:45