* dvc: ignore errors on protect/set_exec (#5335)
Both of these are not fatal and could safely be ignored.
Fixes#5255
(cherry picked from commit 434cd305e9)
* tests: upgrade azurite and enable loose mode
Backport of #5272
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* S3: Allow authentication using session token
* Add a little test
(cherry picked from commit 5374880220)
Co-authored-by: Batuhan Taskaya <[email protected]>
* Skip paramdependency for vars
Previously, `configs` which was a list of PathInfo was being used
to check if the `vars` should be skipped or not.
But, the file is stored as a string that made that check redundant
which might have caused #5117 as they were being stored twice,
once as a expanded paramdependency and next as a flattened dict
from vars, which later during `flatten`, it might have panicked
from that duplication.
* Use defaultdict
* change add_vars to collect_vars
(cherry picked from commit dcce3a31a9)
Co-authored-by: Saugat Pachhai <[email protected]>
* repro: accept multiple targets (#5111)
Currently, we iterate over targets manually in CLI, which results in a
complicated code and a DAG collection/check overhead. With this PR we
collect the DAG once, figure out all the needed targets within it and
just reproduce them in one pass.
(cherry picked from commit aec8d65791)
# Conflicts:
# dvc/command/base.py
# dvc/command/experiments.py
# dvc/command/repro.py
# tests/func/test_repro_multistage.py
* resolve conflicts
* resolve test conflicts
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* fix making too many ssh connections when pushing run cache (#4732)
* reformat
* generator should exit on GeneratorExit exception
* Update dvc/tree/ssh/__init__.py
* pool: close on Exception and not BaseException
Co-authored-by: Tomasz Łakota <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
(cherry picked from commit 6e92352001)
Co-authored-by: peper0 <[email protected]>
Currently we don't handle bad name well:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: unexpected error - not enough values to unpack (expected 2, got 1)
```
With this PR, we check name to conform to either `remote.name.option` or
`section.option`, so we'll give a more friendly:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: argument name: name argument should look like remote.name.option or section.option
usage: dvc config [-h] [--global | --system | --local] [-q | -v] [-u] [-l] [name] [value]
Get or set config options.
Documentation: <https://man.dvc.org/config>
positional arguments:
name Option name (remote.name.option or section.option).
value Option value.
optional arguments:
-h, --help show this help message and exit
--global Use global config.
--system Use system config.
--local Use local config.
-q, --quiet Be quiet.
-v, --verbose Be verbose.
-u, --unset Unset option.
-l, --list List all defined config values.
```
* experiments: commit to refs/exps/... instead of branch heads
* update exp show
* restore checkpoint_resume kwarg
* refactor new() to use experiments stash
* experiments: use refspecs to populate and collect executors
* experiments: use new stash and merge on exp checkout
* experiments: fix executor chdir
* experiments: update run and show
* skip checkout after reproduce
* checkpoints: use experiments refspecs
* check if workspace was stashed before applying
* exp show: unify checkpoint/standalone experiment collection
* git: move iter_refs to backend
* update for scm changes
* experiments: replace exp checkout in favor of exp apply
* git: handle rev_parse errors in resolve_commit
* update refs containing and baseline check
* fix sigint handling for checkpoints
* update checkpoint exists behavior
* fix checkpoint baseline behavior
* update tests
* experiments: update gc
* update show tests
* fix type checking errors
* update diff
* scm: add diff, describe
* experiments: generate human readable names
* exp run/resume: allow user specified exp name with -n/--name
* update tests
* experiments: add `exp branch` for promoting experiment to full branch
* fix test
* fix missing file
* experiments: explicitly close scm instances before cleaning up tmp dir
* git: add support for dulwich Stash
* git: start adding support for custom refs
* git: add Stash and helper context managers for preserving
workspace state
* scm: cleanup refs/stash and add push_refspec
* tests: add tests for git refs and stash
* fix push_refspec
* tests: add tests for git push_refspec
* scm: add git fetch-refspecs
* add test for fetch refspecs
* scm: add test for git refs containing
* remove extraneous log msgs
* fix typing issues
* stash: only try to push if workspace is dirty
* update dulwich to 0.20.14
* scm: move dulwich into scm.backend
* move stash internals to backends
* scm: move git code into dvc/scm/git/... module
* git: use ABC for BaseGitBackend
* fix rebase issue
* git: fix refspec tests on windows
- git file:// URLs need posix style paths
* git: move get_refs_containing to backend
* scm: add callback for handling diverged refs during fetch
Having that logic in `create_stage` is unnatural and also has some
side-effects for unsuspecting users. For example, it will check the
filesystem for existing dvcfile and will try to load a corresponding old
stage. I've adjusted it with minimal changes to better reflect the logic
behind it.
Pre-requisite for #4907
* repro: support regex/foreach-group to run at once
Fixes#4912Fixes#4886Fixes#4958
* Use `tree.isdir` rather than `os.path.isdir`
* Use glob rather than regex
* Update dvc/command/repro.py
* s/regex/glob
* disable glob on `collect_granular`
There's no need for a glob here
* add tests for `collect` and `collect_granular`
* s3: use mocked s3 instead of the real one
Not removing the fixtures just yet
* Update moto
Tests was failing, because of a change in `responses` package, which
`moto` was mocking. This was fixed on
https://github.com/spulec/moto/pull/3466 which is only available on
`dev122` release.
* Use mocked s3 in checkout tests
`Azure` tests are flaky for some reason. When I investigated one
of the job [1], something seemed wrong with the Azurite or
the client lib.
On `AzureTree.blob_service`, we do the following to check if the
container exists:
```python
try:
container_client.get_container_properties()
except ResourceNotFoundError:
container_client.create_container()
```
When looking at the test log, I found out that the
`get_container_properties` was failing with
`ErrorCode:ContainerNotFound` whereas `create_container`
successively was failing with `ErrorCode:ContainerAlreadyExists`.
`create_container` is correct, but `get_container_properties` is
incorrect, as many of the Azure tests have already run successfully
before.
So, either the Azurite has/had bug or the client lib has a bug.
Anyway, we'll find out after updating the Azurite.
[1]: https://github.com/iterative/dvc/runs/1448864706#step:7:4029
Casper da Costa-Luis
Paweł Redzyński
Artem Dudarev
Jorge Orpinel