experimental: support connection strings for databases (#10125)

* experimental: support connection strings for databases using connectorx

This adds an experimental support for connector strings using connectorx.

User has to provide config for the connection strings with:

Example:
```console
dvc config --local db_connection.pg postgres://user:pass@localhost:5432/database
```

And then, they can import using:
```console
dvc import-db --sql 'query' --conn pg
```

Check supported databases and how to configure in https://sfu-db.github.io/connector-x/databases.html.
This `db_connection` config is subject to change.

connectorx still has the same problem with dbt as before: no support for streaming,
but it'll be a bit more simpler to configure with the connection strings.

* add pandas to deps

* sqlalchemy

* dbt streaming sql

* fix test

* remove connectorx

* remove check in set

* use mode='wb' for serialization
This commit is contained in:
skshetry
2023-12-03 19:23:09 +05:45
committed by GitHub
parent 7c61463d2d
commit 8768b50aed
18 changed files with 673 additions and 331 deletions
+3
View File
@@ -77,6 +77,7 @@ gs = ["dvc-gs==2.22.1"]
hdfs = ["dvc-hdfs==2.19"]
lint = [
"mypy==1.7.0",
"pandas-stubs",
"types-colorama",
"types-psutil",
"types-pyinstaller",
@@ -111,6 +112,8 @@ tests = [
'pywin32>=225; sys_platform == "win32"', # optional test dependency
'tzdata; sys_platform == "win32"', # for testing with celery
"dbt-sqlite", # optional dependency for testing `import-db`
"sqlalchemy>=1,<3", # optional dependency for `import-db`
"pandas>=1",
]
webdav = ["dvc-webdav==2.19.1"]
webhdfs = ["dvc-webhdfs==2.19"]