* experimental: support connection strings for databases using connectorx
This adds an experimental support for connector strings using connectorx.
User has to provide config for the connection strings with:
Example:
```console
dvc config --local db_connection.pg postgres://user:pass@localhost:5432/database
```
And then, they can import using:
```console
dvc import-db --sql 'query' --conn pg
```
Check supported databases and how to configure in https://sfu-db.github.io/connector-x/databases.html.
This `db_connection` config is subject to change.
connectorx still has the same problem with dbt as before: no support for streaming,
but it'll be a bit more simpler to configure with the connection strings.
* add pandas to deps
* sqlalchemy
* dbt streaming sql
* fix test
* remove connectorx
* remove check in set
* use mode='wb' for serialization
We should not register a pytest plugin automaticallly, as we may
have fixtures that might override user's fixtures. Also, it might
raise import errors as in https://github.com/iterative/dvc/issues/10118.
We can still use this as a plugin ourselves by setting the following
in `conftest.py` file.
```python
pytest_plugins = ['dvc.testing.plugin']
```
Previously we used recorded index which had stub entries
for missing/failed outputs, but now we actually collect
everything ourselves and those are missing.
* remove pytest-virtualenv
It does not look like it's being actively maintained.
* force virtualenv
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
__name__ looks something like `dvc.commands.diff`, but if we are already instantiating
root logger with `dvc`, it adds up to create something like `dvc.dvc.commands.diff`.
We should be using a root logger here instead.
* use logger.getChild from `dvc.log` instead of `logging.getLogger`
I got tired adding `# type: ignore[attr-defined]` while using `logger.trace`.
So, now you need to import logger from `from dvc.log import logger` and
create a child logger using `logger.getChild(__name__)`. `logger.getChild`
will return a logger type-hinted as a logger having a `trace` method.
So, from now on, instead of:
```python
import logging
logger = logging.getogger(__name__)
```
you'll have to do following:
```python
from dvc.log import logger
logger = logger.getChild(__name__)
```
* remove all dvc imports from dvc.logger
* use Tqdm back
This commit adds the `key` property to the templaing context of `matrix`
stages. Similarly to `foreach` stages, the `key` is available using
`${key}` substitution.
Fixes#10043
See https://docs.python.org/3/library/sys.html#sys.__stdin__.
> [!Note]
> Under some conditions stdin, stdout and stderr as well as the original values __stdin__, __stdout__ and __stderr__ can be None. It is usually the case for Windows GUI apps that aren’t connected to a console and Python apps started with pythonw.
This is a best-effort solution. External libraries (eg: `tqdm`) and future code that is unaware of this stuff may still fail.
As a console application, `stdout`/`stderr` being `None` is not really something that we should support in general.
Regarding the failure of the daemon, it has been fixed so that it'll always have a stream attached (if not, it falls back to
/dev/null). Today, this is a problem in daemon only if the main process is run with no streams attached, which is not
something that we support or go on length to support.
It is tricky to write tests for this case, and I could not come up with one. Also decided not to test daemon as it should no longer be a problem.
skshetry
Peter Rowlands
Ruslan Kuprieiev
Ivan Shcheklein
dependabot[bot]andRuslan Kuprieiev
dberenbaum
Piotr Bielak