python/cpython#113829 changed `os.path.isabs` on Windows to not consider a path starting a single backslash to be an absolute path.
We do a lot of naive `posixpath` to `ntpath` conversions, and it broke DVC in multiple places. DVC is likely broken in many other places, and these bugs are difficult to identify.
In this commit, I have tried to fix in the places where the tests failed. And mostly by trying to imitate pre-3.13 behaviour.
Some of the changes may not be correct but keeps the old behaviour in place.
Also re-enabled the CI for all Python versions supported for Windows.
params/metrics: assume file when dir expansion raises error
Closes https://github.com/iterative/dvc/issues/10018.
We have to be careful when expanding paths. This PR fall backs to assume it
as a file when either of `isdir` and `find` raises error due to some mishaps.
The consequence of this assumption happens when we try to read the file.
In that case, if the directory exists, and we happen to try to open it
as a file, dvc will raise `IsADirectoryError`, but that is handled correctly
and won't fail catastrophically like this did.
`dvc add` command incorrectly raises a `DoesNotExistError` when a
broken symlink exists in an output directory, and the target name
is same as the directory's name.
eg: If `data` is an output, and is the command is invoked as `dvc add data`
(i.e. no virtual directory operations to perform).
The expected behavior to raise a `FileNotFoundError`.
`DoesNotExistError` should only be raised if the output itself does not
exist.
Related: https://github.com/iterative/dvc/issues/3717
* Revert "limit pytest version to <8.1 (#10334)"
This reverts commit bc95b650a4.
* replace flaky with pytest-rerunfailures
`flaky` project seems to be unmaintained (see box/flaky#192).
`pytest-rerunfailures` seems to be maintained under `pytest-dev`
and tests 5 minor releases of pytest.
Note that this plugin is not compatible with `flaky` plugin.
You will have to uninstall it if it exists.
* ds add: use --dvc/--dvcx/--url flags to accept url
.. instead of relying on `scheme://`
I think this is more simpler from user's perspective, easier to
teach and easier to maintain as we won't have to do url parsing.
I am taking inspiration from `cargo install` has `--git`/`--rev`, etc.
to install from alternate sources.
```console
Options:
--version <VERSION> Specify a version to install
--index <INDEX> Registry index to install from
--registry <REGISTRY> Registry to use
--git <URL> Git URL to install the specified crate from
--branch <BRANCH> Branch to use when installing from git
--tag <TAG> Tag to use when installing from git
--rev <SHA> Specific commit to use when installing from git
--path <PATH> Filesystem path to local crate to install
--root <DIR> Directory to install packages into```
```
The `name` is also changed from an option to an argument.
### Examples
```console
dvc ds add example-get-started --dvc [email protected]:iterative/example-get-started.git
dvc ds add dogs --dvcx dogs
dvc ds add dogs --url s3://bucket/key/path
```
* add --version alias
* add support for tracking remote dataset
This adds support for virtually tracking a
dataset such as dvcx dataset, dataset from remote dvc/git
registries, and cloud-versioned remotes.
This PR introduces two different commands under `ds` namespace:
`add` and `update`.
`dvc ds add` command adds the dataset to `dvc.yaml` and tracks
the sources in `dvc.lock` file. Similarly, `dvc ds update`
updates the sources in `dvc.lock` file.
Example Usage:
```console
dvc ds add --name dogs --url dvcx://dogs # tracking dvcx dataset
# tracking dvc/git registries
dvc ds add --name example --url dvc://[email protected]/iterative/example.git
# cloud-versioned-remotes
dvc ds add --name versioning --url s3://cloud-versioning-demo
```
To update, specify the name.
```console
dvc ds update <name>
```
`dvc ds add` freezes (in the traditional sense of `dvc import/import-url`).
It keeps a "specification" of the dataset in `dvc.yaml` and also freezes
information about the dataset in the `dvc.lock` file. They are kept
inside `datasets` section in both `dvc.yaml` and `dvc.lock` files.
This metadata is used in the pipelines. You can add a dependency to your
stage using `ds://` scheme, followed by the name of the dataset in
`dvc.yaml` file. As it is used in pipelines, the `name` of the dataset
has to be unique in the repository. Different dataset of same names
are not allowed. On `dvc repro`, `dvc` copies the frozen information
about the particular dataset into the `deps` field for the stage in
`dvc.lock`. On successive invocation, `dvc` will compare the information
from the deps field of the lock with the frozen information in the
`datasets` section and decides whether to rerun or not.
As the dataset is frozen, `dvc repro` won't rerun until the dataset is
updated via `dvc update`.
Here are some examples for how `dvc.yaml` looks like:
```yaml
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
- name: dogs2
url: dvcx://dogs@v2
type: dvcx
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
```
```yaml
schema: '2.0'
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
rev_lock: df75c16ef61f0772d6e4bb27ba4617b06b4b5398
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
meta:
isdir: true
size: 323919
nfiles: 33
files:
- relpath: myproject/model.pt
meta:
size: 106433
version_id: 5qrtnhnQ4fBzV73kqqK6pMGhTOzd_IPr
etag: 3bc0028677ce6fb65bec8090c248b002
# truncated
```
The pipeline stages keep them in `dataset` section of individual deps.
```console
dvc stage add -n train -d ds://dogs python train.py
cat dvc.yaml
```
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- ds://dogs
```
When it is reproduced, the `dvc.lock` will look something like follows:
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- path: ds://dogs
dataset:
name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
```
* remodeling
* remove non-cloud-versioned remotes
* improve deserializing; handle invalidation
* dvc.api for dataset
* expose the API in the top-level
* add support for tracking remote dataset
This adds support for virtually tracking a
dataset such as dvcx dataset, dataset from remote dvc/git
registries, and cloud-versioned remotes.
This PR introduces two different commands under `ds` namespace:
`add` and `update`.
`dvc ds add` command adds the dataset to `dvc.yaml` and tracks
the sources in `dvc.lock` file. Similarly, `dvc ds update`
updates the sources in `dvc.lock` file.
Example Usage:
```console
dvc ds add --name dogs --url dvcx://dogs # tracking dvcx dataset
# tracking dvc/git registries
dvc ds add --name example --url dvc://[email protected]/iterative/example.git
# cloud-versioned-remotes
dvc ds add --name versioning --url s3://cloud-versioning-demo
```
To update, specify the name.
```console
dvc ds update <name>
```
`dvc ds add` freezes (in the traditional sense of `dvc import/import-url`).
It keeps a "specification" of the dataset in `dvc.yaml` and also freezes
information about the dataset in the `dvc.lock` file. They are kept
inside `datasets` section in both `dvc.yaml` and `dvc.lock` files.
This metadata is used in the pipelines. You can add a dependency to your
stage using `ds://` scheme, followed by the name of the dataset in
`dvc.yaml` file. As it is used in pipelines, the `name` of the dataset
has to be unique in the repository. Different dataset of same names
are not allowed. On `dvc repro`, `dvc` copies the frozen information
about the particular dataset into the `deps` field for the stage in
`dvc.lock`. On successive invocation, `dvc` will compare the information
from the deps field of the lock with the frozen information in the
`datasets` section and decides whether to rerun or not.
As the dataset is frozen, `dvc repro` won't rerun until the dataset is
updated via `dvc update`.
Here are some examples for how `dvc.yaml` looks like:
```yaml
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
- name: dogs2
url: dvcx://dogs@v2
type: dvcx
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
```
```yaml
schema: '2.0'
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
rev_lock: df75c16ef61f0772d6e4bb27ba4617b06b4b5398
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
meta:
isdir: true
size: 323919
nfiles: 33
files:
- relpath: myproject/model.pt
meta:
size: 106433
version_id: 5qrtnhnQ4fBzV73kqqK6pMGhTOzd_IPr
etag: 3bc0028677ce6fb65bec8090c248b002
# truncated
```
The pipeline stages keep them in `dataset` section of individual deps.
```console
dvc stage add -n train -d ds://dogs python train.py
cat dvc.yaml
```
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- ds://dogs
```
When it is reproduced, the `dvc.lock` will look something like follows:
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- path: ds://dogs
dataset:
name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
```
* remodeling
* remove non-cloud-versioned remotes
* improve deserializing; handle invalidation
This was introduced in Python 3.9. The change makes
dict merges 'obvious' and looks cleaner.
I haven't changed `.update()` to `|=` (update operator),
as I find the former to be more readable and "obvious".
The code are wrapped due to us using line-length=79 before, but now some
of the code can easily fit in 88 chars.
The formatter did not change this automatically when we changed line-length,
because of magic trailing commas that gets added when they turn into multiple lines.
Black/ruff wrapping them in parens and splitting them into multiple lines
was making it difficult to read.
* drop Python 3.8
Note that this PR only makes typing related changes.
Other changes can be done once this gets merged, they
are relatively few in numbers.
* bump build python version
* add typing_extensions to lint group
* experimental: support connection strings for databases using connectorx
This adds an experimental support for connector strings using connectorx.
User has to provide config for the connection strings with:
Example:
```console
dvc config --local db_connection.pg postgres://user:pass@localhost:5432/database
```
And then, they can import using:
```console
dvc import-db --sql 'query' --conn pg
```
Check supported databases and how to configure in https://sfu-db.github.io/connector-x/databases.html.
This `db_connection` config is subject to change.
connectorx still has the same problem with dbt as before: no support for streaming,
but it'll be a bit more simpler to configure with the connection strings.
* add pandas to deps
* sqlalchemy
* dbt streaming sql
* fix test
* remove connectorx
* remove check in set
* use mode='wb' for serialization
We should not register a pytest plugin automaticallly, as we may
have fixtures that might override user's fixtures. Also, it might
raise import errors as in https://github.com/iterative/dvc/issues/10118.
We can still use this as a plugin ourselves by setting the following
in `conftest.py` file.
```python
pytest_plugins = ['dvc.testing.plugin']
```
* remove pytest-virtualenv
It does not look like it's being actively maintained.
* force virtualenv
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
__name__ looks something like `dvc.commands.diff`, but if we are already instantiating
root logger with `dvc`, it adds up to create something like `dvc.dvc.commands.diff`.
We should be using a root logger here instead.
* use logger.getChild from `dvc.log` instead of `logging.getLogger`
I got tired adding `# type: ignore[attr-defined]` while using `logger.trace`.
So, now you need to import logger from `from dvc.log import logger` and
create a child logger using `logger.getChild(__name__)`. `logger.getChild`
will return a logger type-hinted as a logger having a `trace` method.
So, from now on, instead of:
```python
import logging
logger = logging.getogger(__name__)
```
you'll have to do following:
```python
from dvc.log import logger
logger = logger.getChild(__name__)
```
* remove all dvc imports from dvc.logger
* use Tqdm back
skshetry
