* experimental: support connection strings for databases using connectorx
This adds an experimental support for connector strings using connectorx.
User has to provide config for the connection strings with:
Example:
```console
dvc config --local db_connection.pg postgres://user:pass@localhost:5432/database
```
And then, they can import using:
```console
dvc import-db --sql 'query' --conn pg
```
Check supported databases and how to configure in https://sfu-db.github.io/connector-x/databases.html.
This `db_connection` config is subject to change.
connectorx still has the same problem with dbt as before: no support for streaming,
but it'll be a bit more simpler to configure with the connection strings.
* add pandas to deps
* sqlalchemy
* dbt streaming sql
* fix test
* remove connectorx
* remove check in set
* use mode='wb' for serialization
* use logger.getChild from `dvc.log` instead of `logging.getLogger`
I got tired adding `# type: ignore[attr-defined]` while using `logger.trace`.
So, now you need to import logger from `from dvc.log import logger` and
create a child logger using `logger.getChild(__name__)`. `logger.getChild`
will return a logger type-hinted as a logger having a `trace` method.
So, from now on, instead of:
```python
import logging
logger = logging.getogger(__name__)
```
you'll have to do following:
```python
from dvc.log import logger
logger = logger.getChild(__name__)
```
* remove all dvc imports from dvc.logger
* use Tqdm back
This commit adds the `key` property to the templaing context of `matrix`
stages. Similarly to `foreach` stages, the `key` is available using
`${key}` substitution.
Fixes#10043
* use popen to run updater/analytics in detached mode
subprocess does not provide a way to avoid a zombie process. Not to confuse with orphaned processes, which will be `wait`-ed by `pid 1`.
It might not work for long-running processes, but I don't think zombie processes are going to be a problem for us, as the parent process immediately exits after spawning a child process.
Even in case of `queue` based experiment runner, parent will exit before than the child process. So I don't think it's a requirement here. But it does feel wrong to leak the processes, so I have to sleep on that.
---
This PR:
1. makes `daemonized` process inherit the logging verbosity, so if the parent process used `-vv`, the daemonized process will run the same way.
2. Add `DVC_DAEMON_LOGFILE` envvar, that the daemon will log to the file. By default, the verbosity is the same as running parent processes. Without `DVC_DAEMON_LOGFILE`, the output will be redirected to `/dev/null`.
3. Adds debug/trace for updater/analytics.
I hope by doing this, we can make our life easier on debugging and make daemon more transparent/observable.
* add tests
* fixes
* use kwargs
* pass all os.environ with some modified
* exp show: show metrics that include separator
* sort using partition instead of lpartition
* handle ':' in file and metric names
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix sorting with leading :
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* update msg for multiple overlapping outputs
* update msg for multiple overlapping outputs
* fix type checking
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use sets of stages to fix mypy errors
* fix tests/func/test_add.py::test_try_adding_multiple_overlaps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] pre-commit autoupdate
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.0.282 → v0.0.285](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.282...v0.0.285)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix ruff issues
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
* use _describe in exp list
* fix pre-commit checks
* simplistic fix
* single _describe, too complex
* slightly simpler
* even simpler
* simple, but with regex
* less substitutions
* describe public
* name handling in UI (tests need updating)
* fixed old tests
* lazy imports
---------
Co-authored-by: Tibor Mach <[email protected]>
* hydra: enable config modules
Lets the user specify Hydra configs in installed Python modules as well
as inside directories. Adds `hydra.config_module` to `.dvc/config`.
Fixes#9740
old tests green
tests for config modules
* hydra: fix linter errors
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* hydra: fix mypy error
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* hydra: fix tests
* Update dvc/utils/hydra.py
Co-authored-by: David de la Iglesia Castro <[email protected]>
* address comments
* convince mypy not to worry
* split long line
* fix test
* check for raised error in test
---------
Co-authored-by: Dom Miketa <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David de la Iglesia Castro <[email protected]>
Ruslan Kuprieiev
skshetry
Dave Berenbaum
Peter Rowlands
Ivan Shcheklein
Piotr Bielak
dberenbaum
Heinz-Alexander Fuetterer
Brady Johnson
David de la Iglesia Castro
Matt Seddon
Tibor MachandTibor Mach
