* Prioritize DVC_SITE_CACHE_DIR over core.site_cache_dir
* Add tests for proper prioritization of env var and config
* Tests require python-3.9 compatible typing, use Optional
* Remove comment
* Fixup
* DVC_SITE_CACHE_DIR must be sent first, deal with platformdirs afterwards
* Test should check default behaviour when no env var is set
* Add test: site_cache_dir() must return the env var if set
Clear logs before assertions so that logs from other tests do not interfere with the test.
Some destructors may also run in between due to GC, so hopefully this will fix that issue too,
or reduce the chances of it happening.
See
https://github.com/iterative/dvc/actions/runs/14873217814/job/41775314725#step:6:97.
```console
> assert first(caplog.messages) == expected_message
E assert 'RC: discarding all clients' == "Output 'path... 'stage.dvc'."
E
E + RC: discarding all clients
E - Output 'path'(stage: 'stage.dvc') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
E - You can also use `dvc commit stage.dvc` to associate existing 'path' with stage: 'stage.dvc'.
```
python/cpython#113829 changed `os.path.isabs` on Windows to not consider a path starting a single backslash to be an absolute path.
We do a lot of naive `posixpath` to `ntpath` conversions, and it broke DVC in multiple places. DVC is likely broken in many other places, and these bugs are difficult to identify.
In this commit, I have tried to fix in the places where the tests failed. And mostly by trying to imitate pre-3.13 behaviour.
Some of the changes may not be correct but keeps the old behaviour in place.
Also re-enabled the CI for all Python versions supported for Windows.
* Add keep_selected parameter, and corresponding code to keep only the selected exps (and remove all the other ones)
* test keep_selected_by_name
* test keep_selected_by_rev
* test keep_selected multiple, by name
* test keep all by name
* test keep by rev, with num=2
* added option to cli
* refactoring to meet pr needs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fixed test_experiments to add keep_selected=False to remove tests
* rename parameter to match cli option
* follow the normal path, then invert the selection before removing
* fixed tests for list ordering + fixed test with non existent name, it didn't make sense to delete everything if an exp name did not exist
* changed cli option comment
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fixed typing issue
* updated parameter name
* removed handling queued experiments (since --queue would remove them all)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* code simplification, added __eq__ and __hash__ to be able to compare ExpRefs, updated and parametrized tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fixed linting issues
* - --keep and --queue together raise an InvalidArgumentError
- added a test to check if the error is raised
- fixed CLI message
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* re-run gh tests. Some tests which did not involve my changes started failing while they were passing fine before.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio login set auto_push + add logs for the user
* set a separate global config dir for studio login tests
The `isolate` fixture sets a separate, _isolated_ global config directory.
But it is shared among other tests. So any mutations to it affects other
tests too.
We should force `studio login` tests to use a separate tmp_dir for global config.
* cleanup message
---------
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
* ds add: use --dvc/--dvcx/--url flags to accept url
.. instead of relying on `scheme://`
I think this is more simpler from user's perspective, easier to
teach and easier to maintain as we won't have to do url parsing.
I am taking inspiration from `cargo install` has `--git`/`--rev`, etc.
to install from alternate sources.
```console
Options:
--version <VERSION> Specify a version to install
--index <INDEX> Registry index to install from
--registry <REGISTRY> Registry to use
--git <URL> Git URL to install the specified crate from
--branch <BRANCH> Branch to use when installing from git
--tag <TAG> Tag to use when installing from git
--rev <SHA> Specific commit to use when installing from git
--path <PATH> Filesystem path to local crate to install
--root <DIR> Directory to install packages into```
```
The `name` is also changed from an option to an argument.
### Examples
```console
dvc ds add example-get-started --dvc [email protected]:iterative/example-get-started.git
dvc ds add dogs --dvcx dogs
dvc ds add dogs --url s3://bucket/key/path
```
* add --version alias
* add support for tracking remote dataset
This adds support for virtually tracking a
dataset such as dvcx dataset, dataset from remote dvc/git
registries, and cloud-versioned remotes.
This PR introduces two different commands under `ds` namespace:
`add` and `update`.
`dvc ds add` command adds the dataset to `dvc.yaml` and tracks
the sources in `dvc.lock` file. Similarly, `dvc ds update`
updates the sources in `dvc.lock` file.
Example Usage:
```console
dvc ds add --name dogs --url dvcx://dogs # tracking dvcx dataset
# tracking dvc/git registries
dvc ds add --name example --url dvc://[email protected]/iterative/example.git
# cloud-versioned-remotes
dvc ds add --name versioning --url s3://cloud-versioning-demo
```
To update, specify the name.
```console
dvc ds update <name>
```
`dvc ds add` freezes (in the traditional sense of `dvc import/import-url`).
It keeps a "specification" of the dataset in `dvc.yaml` and also freezes
information about the dataset in the `dvc.lock` file. They are kept
inside `datasets` section in both `dvc.yaml` and `dvc.lock` files.
This metadata is used in the pipelines. You can add a dependency to your
stage using `ds://` scheme, followed by the name of the dataset in
`dvc.yaml` file. As it is used in pipelines, the `name` of the dataset
has to be unique in the repository. Different dataset of same names
are not allowed. On `dvc repro`, `dvc` copies the frozen information
about the particular dataset into the `deps` field for the stage in
`dvc.lock`. On successive invocation, `dvc` will compare the information
from the deps field of the lock with the frozen information in the
`datasets` section and decides whether to rerun or not.
As the dataset is frozen, `dvc repro` won't rerun until the dataset is
updated via `dvc update`.
Here are some examples for how `dvc.yaml` looks like:
```yaml
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
- name: dogs2
url: dvcx://dogs@v2
type: dvcx
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
```
```yaml
schema: '2.0'
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
rev_lock: df75c16ef61f0772d6e4bb27ba4617b06b4b5398
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
meta:
isdir: true
size: 323919
nfiles: 33
files:
- relpath: myproject/model.pt
meta:
size: 106433
version_id: 5qrtnhnQ4fBzV73kqqK6pMGhTOzd_IPr
etag: 3bc0028677ce6fb65bec8090c248b002
# truncated
```
The pipeline stages keep them in `dataset` section of individual deps.
```console
dvc stage add -n train -d ds://dogs python train.py
cat dvc.yaml
```
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- ds://dogs
```
When it is reproduced, the `dvc.lock` will look something like follows:
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- path: ds://dogs
dataset:
name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
```
* remodeling
* remove non-cloud-versioned remotes
* improve deserializing; handle invalidation
The code are wrapped due to us using line-length=79 before, but now some
of the code can easily fit in 88 chars.
The formatter did not change this automatically when we changed line-length,
because of magic trailing commas that gets added when they turn into multiple lines.
Black/ruff wrapping them in parens and splitting them into multiple lines
was making it difficult to read.
fhausmannandskshetry
Staszek Markowski
Saugat Pachhai (सौगात)
pre-commit-ci[bot]andSaugat Pachhai
B. John Braddock
Samandar MirjalilovandSaugat Pachhai
Rodrigo Goya
Raphaël Michelandpre-commit-ci[bot]
Ivan Shcheklein
dependabot[bot]
dberenbaum
AlexandreKempfandSaugat Pachhai
Matt Seddon
Matt Seddon
Dave Berenbaum