minor refactors, code comments
Rename interim to definitions
provide location of entry during init for definition
small fixups to set_temporarily
Refactor loop to use dict comprehension
Provide same interface for definitions
Make it work for composite lists
Check for sequence rather than a list
refactor tests
Fixes#4955Fixes#4957Fixes#4963
Currently we don't handle bad name well:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: unexpected error - not enough values to unpack (expected 2, got 1)
```
With this PR, we check name to conform to either `remote.name.option` or
`section.option`, so we'll give a more friendly:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: argument name: name argument should look like remote.name.option or section.option
usage: dvc config [-h] [--global | --system | --local] [-q | -v] [-u] [-l] [name] [value]
Get or set config options.
Documentation: <https://man.dvc.org/config>
positional arguments:
name Option name (remote.name.option or section.option).
value Option value.
optional arguments:
-h, --help show this help message and exit
--global Use global config.
--system Use system config.
--local Use local config.
-q, --quiet Be quiet.
-v, --verbose Be verbose.
-u, --unset Unset option.
-l, --list List all defined config values.
```
We try to optimize `tree.exists` calls and probably few others
in that they either look directly into the workspace or,
to the cache without running graph checks. It does not seem
to be possible just to run graph checks on `find_outs_by_path`
due to those optimizations.
So, that's why, the `collect` also does a graph check for this
reason.
Fixes#5027Fixes#4010
* experiments: add utils.exp_commits for iterating over experiment commits
* repo: add support for all_experiments in brancher and used_cache
* git: make all_commits return commits in standard refs instead of --all
* gc: add --all-experiments to preserve cache for experiments
* update gc tests
* repo: support pushing and fetching specific revs internally
* exp push: push cache for experiment by default
* exp pull: pull cache for experiment by default
* fix empty git repo bug
* [WIP] git: use dulwich for add/commit
* dulwich: add is_tracked implementation
* experiments: try to workaround windows cleanup() bug on py < 3.8
* tests: experiments: convert CRLF to LF before comparing params
* trie: updated message for OverlappingOutputsError
the message given by the exception was not verbose enough why this exception is thrown.
now gives the user more information as to why the overlapping paths should be changed.
* test_run_single_stage: fixed test TestShouldRaiseOnOverlappingOutputPaths
the test is now updated to reflect the exception message changes.
Signed-off-by: metalglove <[email protected]>
* git: fix Stash.push args typing
* experiments: handle case where args file was inadvertently committed to
git
* add test for args file being committed to git
* run/repro: show command on dry-run, add ">" to script output
This rollbacks the earlier commit to not show command during
the dry-run, and also prepends ">" in the command script
instead of "$" sign.
* fix tests
* Allow list type in STAGE_DEFINITION scheme and join(cmd) if list
* Remove join in Stage.__init__ and update test
* Loop over commands in cmd_run
* Improve logging of stage.run with detailed by command
* Fix test based on log message
* Update command logging with >
* Add tests for multi lines command
* Update > to $ in stage command log to prevent confusion with > operator
* Fix PR review about test: single quote and read_text
* tests: unify/clarify tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* WIP fixes#4621
* changes to all_branch if condition
* flatten_dict changes
* fix one test
* minor changes
* minor changes
* add precision
* tests and added precision to arguments
Co-authored-by: indh8717 <[email protected]>
* experiments: commit to refs/exps/... instead of branch heads
* update exp show
* restore checkpoint_resume kwarg
* refactor new() to use experiments stash
* experiments: use refspecs to populate and collect executors
* experiments: use new stash and merge on exp checkout
* experiments: fix executor chdir
* experiments: update run and show
* skip checkout after reproduce
* checkpoints: use experiments refspecs
* check if workspace was stashed before applying
* exp show: unify checkpoint/standalone experiment collection
* git: move iter_refs to backend
* update for scm changes
* experiments: replace exp checkout in favor of exp apply
* git: handle rev_parse errors in resolve_commit
* update refs containing and baseline check
* fix sigint handling for checkpoints
* update checkpoint exists behavior
* fix checkpoint baseline behavior
* update tests
* experiments: update gc
* update show tests
* fix type checking errors
* update diff
* scm: add diff, describe
* experiments: generate human readable names
* exp run/resume: allow user specified exp name with -n/--name
* update tests
* experiments: add `exp branch` for promoting experiment to full branch
* fix test
* fix missing file
* experiments: explicitly close scm instances before cleaning up tmp dir
* git: add support for dulwich Stash
* git: start adding support for custom refs
* git: add Stash and helper context managers for preserving
workspace state
* scm: cleanup refs/stash and add push_refspec
* tests: add tests for git refs and stash
* fix push_refspec
* tests: add tests for git push_refspec
* scm: add git fetch-refspecs
* add test for fetch refspecs
* scm: add test for git refs containing
* remove extraneous log msgs
* fix typing issues
* stash: only try to push if workspace is dirty
* update dulwich to 0.20.14
* scm: move dulwich into scm.backend
* move stash internals to backends
* scm: move git code into dvc/scm/git/... module
* git: use ABC for BaseGitBackend
* fix rebase issue
* git: fix refspec tests on windows
- git file:// URLs need posix style paths
* git: move get_refs_containing to backend
* scm: add callback for handling diverged refs during fetch
* repro: support regex/foreach-group to run at once
Fixes#4912Fixes#4886Fixes#4958
* Use `tree.isdir` rather than `os.path.isdir`
* Use glob rather than regex
* Update dvc/command/repro.py
* s/regex/glob
* disable glob on `collect_granular`
There's no need for a glob here
* add tests for `collect` and `collect_granular`
* s3: use mocked s3 instead of the real one
Not removing the fixtures just yet
* Update moto
Tests was failing, because of a change in `responses` package, which
`moto` was mocking. This was fixed on
https://github.com/spulec/moto/pull/3466 which is only available on
`dev122` release.
* Use mocked s3 in checkout tests
* parametrization: prevent dumping parametrized stages
This is a bit hackish, but does the job done.
It feels like we need some unified way to be able to load
and dump, but not sure how. Till then, this dataclass
will be helpful in communicating between those.
* s/Meta/RawData
* Update dvc/stage/loader.py
Some of the tests are run by default, other requires
`--enable-{remote}` to run. The remotes that are run by default
also have a `--disable-{remote}` flag. All of the remotes can be
enabled using `--all` flag.
To use this functionality in the fixture,
`test_config` fixture has to be used, and requires to call
`test_config.requires(remote)`.
In an individual test, you can simply use `pytest.mark.remote`
to make them skip or run.
The schema is changed of `vars` to accomodate the both use of
`imports` and setting variables locally.
Now:
```
vars:
- params.yaml
- foo: foo
bar: bar
```
Also, same distinction occurs for the stages.
* dvc: include cache urls in info output.
Show cache urls for external and remote cache in the info output
to assist in diagnosis of issues.
Fixes#4574.
* Remove unnecessary constant check for generator expression.
* Add remotes and caches in version command for better diagnosis.
* Fix flaky unit test to be runtime independent by using regex search.
* Use Cache.by_scheme() for enumerating supported caches.
* Add get_cloud_schemes routine for retrieval of configured remote schemes.
Add tests for local remote and remote:// scheme for info.
* Cleanup unused code.
* info: refactor remote parsing
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Outs desc will be used by viewer right away. Stage desc is for people to
start using now, but we won't be using it it viewer until DAG support is
added there.
Saugat Pachhai
Ruslan Kuprieiev
Batuhan Taskaya
GaoandSaugat Pachhai
Peter Rowlands (변기호)
metalglove
Clément WalterandRuslan Kuprieiev
Indhu Kumarandindh8717
Artem Dudarev
Sami Jawhar
Arun MahapatraandRuslan Kuprieiev
Paweł Redzyński