* ds add: use --dvc/--dvcx/--url flags to accept url
.. instead of relying on `scheme://`
I think this is more simpler from user's perspective, easier to
teach and easier to maintain as we won't have to do url parsing.
I am taking inspiration from `cargo install` has `--git`/`--rev`, etc.
to install from alternate sources.
```console
Options:
--version <VERSION> Specify a version to install
--index <INDEX> Registry index to install from
--registry <REGISTRY> Registry to use
--git <URL> Git URL to install the specified crate from
--branch <BRANCH> Branch to use when installing from git
--tag <TAG> Tag to use when installing from git
--rev <SHA> Specific commit to use when installing from git
--path <PATH> Filesystem path to local crate to install
--root <DIR> Directory to install packages into```
```
The `name` is also changed from an option to an argument.
### Examples
```console
dvc ds add example-get-started --dvc [email protected]:iterative/example-get-started.git
dvc ds add dogs --dvcx dogs
dvc ds add dogs --url s3://bucket/key/path
```
* add --version alias
The code are wrapped due to us using line-length=79 before, but now some
of the code can easily fit in 88 chars.
The formatter did not change this automatically when we changed line-length,
because of magic trailing commas that gets added when they turn into multiple lines.
Black/ruff wrapping them in parens and splitting them into multiple lines
was making it difficult to read.
* drop Python 3.8
Note that this PR only makes typing related changes.
Other changes can be done once this gets merged, they
are relatively few in numbers.
* bump build python version
* add typing_extensions to lint group
* experimental: support connection strings for databases using connectorx
This adds an experimental support for connector strings using connectorx.
User has to provide config for the connection strings with:
Example:
```console
dvc config --local db_connection.pg postgres://user:pass@localhost:5432/database
```
And then, they can import using:
```console
dvc import-db --sql 'query' --conn pg
```
Check supported databases and how to configure in https://sfu-db.github.io/connector-x/databases.html.
This `db_connection` config is subject to change.
connectorx still has the same problem with dbt as before: no support for streaming,
but it'll be a bit more simpler to configure with the connection strings.
* add pandas to deps
* sqlalchemy
* dbt streaming sql
* fix test
* remove connectorx
* remove check in set
* use mode='wb' for serialization
* use logger.getChild from `dvc.log` instead of `logging.getLogger`
I got tired adding `# type: ignore[attr-defined]` while using `logger.trace`.
So, now you need to import logger from `from dvc.log import logger` and
create a child logger using `logger.getChild(__name__)`. `logger.getChild`
will return a logger type-hinted as a logger having a `trace` method.
So, from now on, instead of:
```python
import logging
logger = logging.getogger(__name__)
```
you'll have to do following:
```python
from dvc.log import logger
logger = logger.getChild(__name__)
```
* remove all dvc imports from dvc.logger
* use Tqdm back
This commit adds the `key` property to the templaing context of `matrix`
stages. Similarly to `foreach` stages, the `key` is available using
`${key}` substitution.
Fixes#10043
* use popen to run updater/analytics in detached mode
subprocess does not provide a way to avoid a zombie process. Not to confuse with orphaned processes, which will be `wait`-ed by `pid 1`.
It might not work for long-running processes, but I don't think zombie processes are going to be a problem for us, as the parent process immediately exits after spawning a child process.
Even in case of `queue` based experiment runner, parent will exit before than the child process. So I don't think it's a requirement here. But it does feel wrong to leak the processes, so I have to sleep on that.
---
This PR:
1. makes `daemonized` process inherit the logging verbosity, so if the parent process used `-vv`, the daemonized process will run the same way.
2. Add `DVC_DAEMON_LOGFILE` envvar, that the daemon will log to the file. By default, the verbosity is the same as running parent processes. Without `DVC_DAEMON_LOGFILE`, the output will be redirected to `/dev/null`.
3. Adds debug/trace for updater/analytics.
I hope by doing this, we can make our life easier on debugging and make daemon more transparent/observable.
* add tests
* fixes
* use kwargs
* pass all os.environ with some modified
skshetry
Matt Seddon
Dave Berenbaum
Ruslan Kuprieievanddberenbaum
Peter Rowlands
Ivan Shcheklein
Piotr Bielak
Heinz-Alexander Fuetterer
Brady Johnson