Currently, we iterate over targets manually in CLI, which results in a
complicated code and a DAG collection/check overhead. With this PR we
collect the DAG once, figure out all the needed targets within it and
just reproduce them in one pass.
* fix making too many ssh connections when pushing run cache (#4732)
* reformat
* generator should exit on GeneratorExit exception
* Update dvc/tree/ssh/__init__.py
* pool: close on Exception and not BaseException
Co-authored-by: Tomasz Łakota <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
minor refactors, code comments
Rename interim to definitions
provide location of entry during init for definition
small fixups to set_temporarily
Refactor loop to use dict comprehension
Provide same interface for definitions
Make it work for composite lists
Check for sequence rather than a list
refactor tests
Fixes#4955Fixes#4957Fixes#4963
Currently we don't handle bad name well:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: unexpected error - not enough values to unpack (expected 2, got 1)
```
With this PR, we check name to conform to either `remote.name.option` or
`section.option`, so we'll give a more friendly:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: argument name: name argument should look like remote.name.option or section.option
usage: dvc config [-h] [--global | --system | --local] [-q | -v] [-u] [-l] [name] [value]
Get or set config options.
Documentation: <https://man.dvc.org/config>
positional arguments:
name Option name (remote.name.option or section.option).
value Option value.
optional arguments:
-h, --help show this help message and exit
--global Use global config.
--system Use system config.
--local Use local config.
-q, --quiet Be quiet.
-v, --verbose Be verbose.
-u, --unset Unset option.
-l, --list List all defined config values.
```
We try to optimize `tree.exists` calls and probably few others
in that they either look directly into the workspace or,
to the cache without running graph checks. It does not seem
to be possible just to run graph checks on `find_outs_by_path`
due to those optimizations.
So, that's why, the `collect` also does a graph check for this
reason.
Fixes#5027Fixes#4010
* experiments: add utils.exp_commits for iterating over experiment commits
* repo: add support for all_experiments in brancher and used_cache
* git: make all_commits return commits in standard refs instead of --all
* gc: add --all-experiments to preserve cache for experiments
* update gc tests
* repo: support pushing and fetching specific revs internally
* exp push: push cache for experiment by default
* exp pull: pull cache for experiment by default
* fix empty git repo bug
* [WIP] git: use dulwich for add/commit
* dulwich: add is_tracked implementation
* experiments: try to workaround windows cleanup() bug on py < 3.8
* tests: experiments: convert CRLF to LF before comparing params
* trie: updated message for OverlappingOutputsError
the message given by the exception was not verbose enough why this exception is thrown.
now gives the user more information as to why the overlapping paths should be changed.
* test_run_single_stage: fixed test TestShouldRaiseOnOverlappingOutputPaths
the test is now updated to reflect the exception message changes.
Signed-off-by: metalglove <[email protected]>
* git: remove scm from backends, make backends configurable
* git: implement get/set/remove ref in gitpython
* git: support reflog --rewrite for stash
* tests: support testing individual backends in unit/scm/test_git
* stash: create reflog if it does not exist for gitpython backend
* git: fix Stash.push args typing
* experiments: handle case where args file was inadvertently committed to
git
* add test for args file being committed to git
* run/repro: show command on dry-run, add ">" to script output
This rollbacks the earlier commit to not show command during
the dry-run, and also prepends ">" in the command script
instead of "$" sign.
* fix tests
* Allow list type in STAGE_DEFINITION scheme and join(cmd) if list
* Remove join in Stage.__init__ and update test
* Loop over commands in cmd_run
* Improve logging of stage.run with detailed by command
* Fix test based on log message
* Update command logging with >
* Add tests for multi lines command
* Update > to $ in stage command log to prevent confusion with > operator
* Fix PR review about test: single quote and read_text
* tests: unify/clarify tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev
Peter Rowlands (변기호)
João Rafael
Sandeep MistryandRuslan Kuprieiev
Saugat Pachhai
Batuhan Taskaya
GaoandSaugat Pachhai
Johnny Chen
metalglove
OrangeCubeandGary Yuan
Victor
Clément WalterandRuslan Kuprieiev