* add auto_push for experiment on run and on save.
add config.exp.auto_push and config.exp.git_remote
* add func tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* separate auto_push behaviour out of commit
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
* Revert "limit pytest version to <8.1 (#10334)"
This reverts commit bc95b650a4.
* replace flaky with pytest-rerunfailures
`flaky` project seems to be unmaintained (see box/flaky#192).
`pytest-rerunfailures` seems to be maintained under `pytest-dev`
and tests 5 minor releases of pytest.
Note that this plugin is not compatible with `flaky` plugin.
You will have to uninstall it if it exists.
* ds add: use --dvc/--dvcx/--url flags to accept url
.. instead of relying on `scheme://`
I think this is more simpler from user's perspective, easier to
teach and easier to maintain as we won't have to do url parsing.
I am taking inspiration from `cargo install` has `--git`/`--rev`, etc.
to install from alternate sources.
```console
Options:
--version <VERSION> Specify a version to install
--index <INDEX> Registry index to install from
--registry <REGISTRY> Registry to use
--git <URL> Git URL to install the specified crate from
--branch <BRANCH> Branch to use when installing from git
--tag <TAG> Tag to use when installing from git
--rev <SHA> Specific commit to use when installing from git
--path <PATH> Filesystem path to local crate to install
--root <DIR> Directory to install packages into```
```
The `name` is also changed from an option to an argument.
### Examples
```console
dvc ds add example-get-started --dvc [email protected]:iterative/example-get-started.git
dvc ds add dogs --dvcx dogs
dvc ds add dogs --url s3://bucket/key/path
```
* add --version alias
* add support for tracking remote dataset
This adds support for virtually tracking a
dataset such as dvcx dataset, dataset from remote dvc/git
registries, and cloud-versioned remotes.
This PR introduces two different commands under `ds` namespace:
`add` and `update`.
`dvc ds add` command adds the dataset to `dvc.yaml` and tracks
the sources in `dvc.lock` file. Similarly, `dvc ds update`
updates the sources in `dvc.lock` file.
Example Usage:
```console
dvc ds add --name dogs --url dvcx://dogs # tracking dvcx dataset
# tracking dvc/git registries
dvc ds add --name example --url dvc://[email protected]/iterative/example.git
# cloud-versioned-remotes
dvc ds add --name versioning --url s3://cloud-versioning-demo
```
To update, specify the name.
```console
dvc ds update <name>
```
`dvc ds add` freezes (in the traditional sense of `dvc import/import-url`).
It keeps a "specification" of the dataset in `dvc.yaml` and also freezes
information about the dataset in the `dvc.lock` file. They are kept
inside `datasets` section in both `dvc.yaml` and `dvc.lock` files.
This metadata is used in the pipelines. You can add a dependency to your
stage using `ds://` scheme, followed by the name of the dataset in
`dvc.yaml` file. As it is used in pipelines, the `name` of the dataset
has to be unique in the repository. Different dataset of same names
are not allowed. On `dvc repro`, `dvc` copies the frozen information
about the particular dataset into the `deps` field for the stage in
`dvc.lock`. On successive invocation, `dvc` will compare the information
from the deps field of the lock with the frozen information in the
`datasets` section and decides whether to rerun or not.
As the dataset is frozen, `dvc repro` won't rerun until the dataset is
updated via `dvc update`.
Here are some examples for how `dvc.yaml` looks like:
```yaml
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
- name: dogs2
url: dvcx://dogs@v2
type: dvcx
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
```
```yaml
schema: '2.0'
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
rev_lock: df75c16ef61f0772d6e4bb27ba4617b06b4b5398
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
meta:
isdir: true
size: 323919
nfiles: 33
files:
- relpath: myproject/model.pt
meta:
size: 106433
version_id: 5qrtnhnQ4fBzV73kqqK6pMGhTOzd_IPr
etag: 3bc0028677ce6fb65bec8090c248b002
# truncated
```
The pipeline stages keep them in `dataset` section of individual deps.
```console
dvc stage add -n train -d ds://dogs python train.py
cat dvc.yaml
```
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- ds://dogs
```
When it is reproduced, the `dvc.lock` will look something like follows:
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- path: ds://dogs
dataset:
name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
```
* remodeling
* remove non-cloud-versioned remotes
* improve deserializing; handle invalidation
* dvc.api for dataset
* expose the API in the top-level
* add support for tracking remote dataset
This adds support for virtually tracking a
dataset such as dvcx dataset, dataset from remote dvc/git
registries, and cloud-versioned remotes.
This PR introduces two different commands under `ds` namespace:
`add` and `update`.
`dvc ds add` command adds the dataset to `dvc.yaml` and tracks
the sources in `dvc.lock` file. Similarly, `dvc ds update`
updates the sources in `dvc.lock` file.
Example Usage:
```console
dvc ds add --name dogs --url dvcx://dogs # tracking dvcx dataset
# tracking dvc/git registries
dvc ds add --name example --url dvc://[email protected]/iterative/example.git
# cloud-versioned-remotes
dvc ds add --name versioning --url s3://cloud-versioning-demo
```
To update, specify the name.
```console
dvc ds update <name>
```
`dvc ds add` freezes (in the traditional sense of `dvc import/import-url`).
It keeps a "specification" of the dataset in `dvc.yaml` and also freezes
information about the dataset in the `dvc.lock` file. They are kept
inside `datasets` section in both `dvc.yaml` and `dvc.lock` files.
This metadata is used in the pipelines. You can add a dependency to your
stage using `ds://` scheme, followed by the name of the dataset in
`dvc.yaml` file. As it is used in pipelines, the `name` of the dataset
has to be unique in the repository. Different dataset of same names
are not allowed. On `dvc repro`, `dvc` copies the frozen information
about the particular dataset into the `deps` field for the stage in
`dvc.lock`. On successive invocation, `dvc` will compare the information
from the deps field of the lock with the frozen information in the
`datasets` section and decides whether to rerun or not.
As the dataset is frozen, `dvc repro` won't rerun until the dataset is
updated via `dvc update`.
Here are some examples for how `dvc.yaml` looks like:
```yaml
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
- name: dogs2
url: dvcx://dogs@v2
type: dvcx
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
```
```yaml
schema: '2.0'
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
rev_lock: df75c16ef61f0772d6e4bb27ba4617b06b4b5398
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
meta:
isdir: true
size: 323919
nfiles: 33
files:
- relpath: myproject/model.pt
meta:
size: 106433
version_id: 5qrtnhnQ4fBzV73kqqK6pMGhTOzd_IPr
etag: 3bc0028677ce6fb65bec8090c248b002
# truncated
```
The pipeline stages keep them in `dataset` section of individual deps.
```console
dvc stage add -n train -d ds://dogs python train.py
cat dvc.yaml
```
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- ds://dogs
```
When it is reproduced, the `dvc.lock` will look something like follows:
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- path: ds://dogs
dataset:
name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
```
* remodeling
* remove non-cloud-versioned remotes
* improve deserializing; handle invalidation
This was introduced in Python 3.9. The change makes
dict merges 'obvious' and looks cleaner.
I haven't changed `.update()` to `|=` (update operator),
as I find the former to be more readable and "obvious".
* repo: site_cache_dir: separate subdir from root_dir
Currently we join scm root_dir with subrepo root dir to have something that would represent that subrepo, but that directory might not exist in reality as we might be using a bare git repo. So we should just treat those as separate things.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update __init__.py
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The code are wrapped due to us using line-length=79 before, but now some
of the code can easily fit in 88 chars.
The formatter did not change this automatically when we changed line-length,
because of magic trailing commas that gets added when they turn into multiple lines.
Black/ruff wrapping them in parens and splitting them into multiple lines
was making it difficult to read.
* drop Python 3.8
Note that this PR only makes typing related changes.
Other changes can be done once this gets merged, they
are relatively few in numbers.
* bump build python version
* add typing_extensions to lint group
skshetry
dependabot[bot]
Matt Seddon
Peter Rowlands
Ruslan Kuprieiev
Matt Seddon
Dave Berenbaum