`data_keys` are similar to `outs`, but they provide more granularity, so we
can use them in `build_data_index` to rebuild index views with filtered
outputs. E.g. when running `dvc diff dir/file`, workspace index should only
have `dir/file` in it.
This also could be used for standalone (non-output) data in the future.
Pre-requisite for #8930
Otherwise when using brancher we end up in a strange situation where even
though we are collecting targets from `gitfs`, we are still using `localfs`
as `self.fs`, which make operations like `self.fs.abspath` in
`collect_granular` return invalid paths.
If we want `add`, `commit` and `move` to work for cloud versioning, option must be always True so it is the same as not having it.
Closes#8828
(We were not passing `merge_versioned=True` in `dvc commit`)
* cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc.
* remove pylint direcitves
* Apply suggestions from code review
* Update pyproject.toml
* build(deps-dev): bump dvc-task from 0.1.9 to 0.1.10
Bumps [dvc-task](https://github.com/iterative/dvc-task) from 0.1.9 to 0.1.10.
- [Release notes](https://github.com/iterative/dvc-task/releases)
- [Commits](https://github.com/iterative/dvc-task/compare/0.1.9...0.1.10)
---
updated-dependencies:
- dependency-name: dvc-task
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <[email protected]>
* remove xfail on json.JSONDecodeError/PermissionError
JSONDecodeError should be fixed by the new release. Next we need to investigate
PermissionError.
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
Cuts down `dvc ls . data/mnist/dataset --recursive` for me from ~37 to ~25
seconds.
Note that this also changes behaviour for ls with a broken symlink, as we
were not able to see it before because of a failing `info()` call, but now
we can follow a normal UNIX ls behaviour more closely.
fix: #8612
For checkpoint experiment in some case users might want to give it a early
stopping to reduce variance. But currently if we interrupt/kill the
experiment it will be marked as failed, and all of the completed
checkpoints will be removed as we cleanup all the running directly after
the process failed.
1. We raise CheckpointKilledError intead of StageCmdFailed error if at
least one checkpoint had been commited.
2. Temp_dir executor will continue collecting data if the Checkpoint
stage was interrupted.
3. Raise warnings if a checkpoint stage was incomplete and the other
stages were not forwarded.
4. Add a new functional test for this
- Add `get_random_exp_name` to utils.
Generates a name (`f"{adjective}-{noun}"`) by randomly chosing pairs of adjetive, noun.
Recieves `scm` and `baseline_rev` and retries the random choice until a non-existing ref is found.
Closes https://github.com/iterative/dvc/issues/8650
skshetry
Ruslan Kuprieiev
Daniele Trifirò
David de la Iglesia Castro
Peter Rowlands
karajan1001
Ivan Shcheklein
dberenbaum