This adds support for custom metadata in outputs under `meta` key.
Also adds `--meta` flag to `add`/`import`/`import-url` that supports
`key=value` values.
We don't have aiohttp support yet, due to which we have to install
without cythonized extension.
Similarly, there has been some changes in how dataclasses disallows some
mutable arguments to be set as a class property, due to which
`hydra.conf` fails to import in 3.11.
See https://github.com/python/cpython/pull/29867. For now, I have
patched the module and avoid importing as a pytest-plugin.
dvc-hdfs/pyarrow is not there yet, we'll usually see a release 2-3
months after official reelase.
dvc-hdfs is skipped in 3.11 in extras_requires. `--set-params` will
return a nice error message if hydra cannot be imported in 3.11. I
decided to only patch that in tests.
Update dvc/commands/plots.py
Co-authored-by: David de la Iglesia Castro <[email protected]>
Update tests/unit/command/test_plots.py
Co-authored-by: David de la Iglesia Castro <[email protected]>
Update dvc/commands/plots.py
Co-authored-by: Paweł Redzyński <[email protected]>
fixed test
add --no-download flag to dvc import-url/dvc update to only create/update .dvc
files without downloading the associated data.
Created .dvc files can be fetched using `dvc pull` or can be
updated using `dvc update --no-download`.
fix: #8014
> ERROR: Invalid experiment '{entry.stash_rev[:7]}'.
This happens when the queue task failed from a scm error.
{"exc_type": "GitMergeError", "exc_message": ["Cannot fast-forward HEAD
to '05100047a341f2fa4a02421289d48f84d8c45e86'"], "exc_module": "dvc.scm"}.
And the failed task neither create a infofile nor create a fail_stash.
1. Didn't raise DvcException if no infofile found for failed tasks.
(success tasks still raise it)
2. Add a new unit test for this.
- Overriding a config value : foo.bar=value
- Appending a config value : +foo.bar=value
- Appending or overriding a config value : ++foo.bar=value
- Removing a config value : ~foo.bar, ~foo.bar=value
See https://hydra.cc/docs/advanced/override_grammar/basic/#modifying-the-config-object
---
Breaking changes:
To modify a list, `foo[0]=bar` must now be passed as `foo.0=bar`.
Modifying a nested list inside a dictionary is not supported by omegaconf.
---
Closes#4883Closes#5868Closes#6129
fix: #8018fix: #8071
1. Add one line to make `exp remove` can remove failed tasks, and will
also delete associated successful tasks
2. Add a new unit test for this behaviour.
3. Solve the unstable test_queue problem.
`--untracked-files` and `--unchanged` coupled with `--granular`
puts a lot of stress on `show_status`. Currently, with those options,
dvc spends ~26s on `_show_status` function.
Unhappy with the above result, I looked into improving the situation with
`console.out`. It did reduce that to 9s but it still felt like a lot.
Then I dropped to good old colorama, which dropped to 3s(term)/1.24(redirected).
I tested with buffering output which dropped this further to
~600ms(term)/~100ms(redirected).
This may seem like premature optimization, but this was preventing me
from figuring out perf issues in granular outputs, and I believe the
user should not be penalize for using feature that we already offer.
fix: #8031
Currently when we use tmp directory to run a experiment, we set
`EXEC_HEAD` , `EXEC_BASELINE` and `EXEC_MERGE` heads locally but forget
to remove it. This cause some problem when we use `exp show`
1. Clean `EXEC_HEAD` , `EXEC_BASELINE` and `EXEC_MERGE` after we pushed
them to tmp_dir
2. Add a unit test for it.
Co-authored-by:
Followed from #7592
1. Seperate remove method to a new file.
2. Add queued, failed, processed flags to remove.
3. Add new unit tests for queue status --queue/fail/success
4. Implement methods to remove done tasks.
5. add some new unit and functional test for celery_queue.remove
6. bump into dvc-task version 0.0.13
Ruslan Kuprieiev
skshetry
Peter Rowlands
Dave Berenbaum
David de la Iglesia Castro
Daniele Trifirò
Ivan Shcheklein
karajan1001
Sami Abdul-Wahid