experimental: support connection strings for databases (#10125)

* experimental: support connection strings for databases using connectorx

This adds an experimental support for connector strings using connectorx.

User has to provide config for the connection strings with:

Example:
```console
dvc config --local db_connection.pg postgres://user:pass@localhost:5432/database
```

And then, they can import using:
```console
dvc import-db --sql 'query' --conn pg
```

Check supported databases and how to configure in https://sfu-db.github.io/connector-x/databases.html.
This `db_connection` config is subject to change.

connectorx still has the same problem with dbt as before: no support for streaming,
but it'll be a bit more simpler to configure with the connection strings.

* add pandas to deps

* sqlalchemy

* dbt streaming sql

* fix test

* remove connectorx

* remove check in set

* use mode='wb' for serialization
This commit is contained in:
skshetry
2023-12-03 19:23:09 +05:45
committed by GitHub
parent 7c61463d2d
commit 8768b50aed
18 changed files with 673 additions and 331 deletions
+1 -1
View File
@@ -96,7 +96,7 @@ def test_config_unset(dvc):
cmd = args.func(args)
assert isinstance(cmd, CmdConfig)
assert args.unset
assert args.name == (False, "section", "option")
assert args.name == (None, "section", "option")
assert args.value == value