* Revert "limit pytest version to <8.1 (#10334)"
This reverts commit bc95b650a4.
* replace flaky with pytest-rerunfailures
`flaky` project seems to be unmaintained (see box/flaky#192).
`pytest-rerunfailures` seems to be maintained under `pytest-dev`
and tests 5 minor releases of pytest.
Note that this plugin is not compatible with `flaky` plugin.
You will have to uninstall it if it exists.
* ds add: use --dvc/--dvcx/--url flags to accept url
.. instead of relying on `scheme://`
I think this is more simpler from user's perspective, easier to
teach and easier to maintain as we won't have to do url parsing.
I am taking inspiration from `cargo install` has `--git`/`--rev`, etc.
to install from alternate sources.
```console
Options:
--version <VERSION> Specify a version to install
--index <INDEX> Registry index to install from
--registry <REGISTRY> Registry to use
--git <URL> Git URL to install the specified crate from
--branch <BRANCH> Branch to use when installing from git
--tag <TAG> Tag to use when installing from git
--rev <SHA> Specific commit to use when installing from git
--path <PATH> Filesystem path to local crate to install
--root <DIR> Directory to install packages into```
```
The `name` is also changed from an option to an argument.
### Examples
```console
dvc ds add example-get-started --dvc [email protected]:iterative/example-get-started.git
dvc ds add dogs --dvcx dogs
dvc ds add dogs --url s3://bucket/key/path
```
* add --version alias
* add support for tracking remote dataset
This adds support for virtually tracking a
dataset such as dvcx dataset, dataset from remote dvc/git
registries, and cloud-versioned remotes.
This PR introduces two different commands under `ds` namespace:
`add` and `update`.
`dvc ds add` command adds the dataset to `dvc.yaml` and tracks
the sources in `dvc.lock` file. Similarly, `dvc ds update`
updates the sources in `dvc.lock` file.
Example Usage:
```console
dvc ds add --name dogs --url dvcx://dogs # tracking dvcx dataset
# tracking dvc/git registries
dvc ds add --name example --url dvc://[email protected]/iterative/example.git
# cloud-versioned-remotes
dvc ds add --name versioning --url s3://cloud-versioning-demo
```
To update, specify the name.
```console
dvc ds update <name>
```
`dvc ds add` freezes (in the traditional sense of `dvc import/import-url`).
It keeps a "specification" of the dataset in `dvc.yaml` and also freezes
information about the dataset in the `dvc.lock` file. They are kept
inside `datasets` section in both `dvc.yaml` and `dvc.lock` files.
This metadata is used in the pipelines. You can add a dependency to your
stage using `ds://` scheme, followed by the name of the dataset in
`dvc.yaml` file. As it is used in pipelines, the `name` of the dataset
has to be unique in the repository. Different dataset of same names
are not allowed. On `dvc repro`, `dvc` copies the frozen information
about the particular dataset into the `deps` field for the stage in
`dvc.lock`. On successive invocation, `dvc` will compare the information
from the deps field of the lock with the frozen information in the
`datasets` section and decides whether to rerun or not.
As the dataset is frozen, `dvc repro` won't rerun until the dataset is
updated via `dvc update`.
Here are some examples for how `dvc.yaml` looks like:
```yaml
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
- name: dogs2
url: dvcx://dogs@v2
type: dvcx
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
```
```yaml
schema: '2.0'
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
rev_lock: df75c16ef61f0772d6e4bb27ba4617b06b4b5398
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
meta:
isdir: true
size: 323919
nfiles: 33
files:
- relpath: myproject/model.pt
meta:
size: 106433
version_id: 5qrtnhnQ4fBzV73kqqK6pMGhTOzd_IPr
etag: 3bc0028677ce6fb65bec8090c248b002
# truncated
```
The pipeline stages keep them in `dataset` section of individual deps.
```console
dvc stage add -n train -d ds://dogs python train.py
cat dvc.yaml
```
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- ds://dogs
```
When it is reproduced, the `dvc.lock` will look something like follows:
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- path: ds://dogs
dataset:
name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
```
* remodeling
* remove non-cloud-versioned remotes
* improve deserializing; handle invalidation
This was introduced in Python 3.9. The change makes
dict merges 'obvious' and looks cleaner.
I haven't changed `.update()` to `|=` (update operator),
as I find the former to be more readable and "obvious".
The code are wrapped due to us using line-length=79 before, but now some
of the code can easily fit in 88 chars.
The formatter did not change this automatically when we changed line-length,
because of magic trailing commas that gets added when they turn into multiple lines.
Black/ruff wrapping them in parens and splitting them into multiple lines
was making it difficult to read.
* drop Python 3.8
Note that this PR only makes typing related changes.
Other changes can be done once this gets merged, they
are relatively few in numbers.
* bump build python version
* add typing_extensions to lint group
* standardise plot data output
* add data required for new anchors to renderer properties
* feed dvc-render updates into API output
* update --json --split api with new data types (send split anchor definitions as strings)
* accomodate zoom and pan anchor
* switch anchor_revs to revs_with_datapoints
* accomodate height and width anchors
* drop terrible idea of holding all data as strings
* fix plot having multiple x fields
* move inferred properties off vega converter class
* Add Studio authentication to DVC
Implemented a new feature to authenticate DVC with Iterative Studio. This includes generating an access token, starting device login, checking if user code is authorized, and handling any request exceptions. Furthermore, the commit creates a new authentication command for DVC. This feature was added to allow DVC users to authenticate with Studio, which is necessary for functionality such as sharing live experiments and notifying Studio about pushed experiments.
* Add authentication validation and error handling to DVC Studio
This commit introduces validation for scopes used in DVC Studio's authentication process. If an unsupported scope is passed, the user will receive an error message listing the invalid scopes and the program will return an error status. Additionally, this commit changes the program's exit behavior when device code authorization fails or authentication process is successful, replacing 'sys.exit' with appropriate returned values.
* Add logout functionality to DVC Studio commands
This commit introduces the logout command for DVC Studio. It allows users to manually log out from DVC Studio via the command line interface. The logout command only removes the user's token if it exists, otherwise, it will print an error message and return an error status. This adds an extra layer of control for the users and enhances the security for the usage of DVC Studio.
* Improve User Authentication process for DVC Studio
Implemented a new 'token' command facilitating users to check their authentication token while operating with DVC Studio. This implementation enhances user experience by providing the ability to validate logged-in status. If not logged in, it rightly prompts an error, ensuring secure access.
* Add unit tests for auth commands
Unit tests are added for the authentication commands used in DVC Studio. The modifications include tests for login with invalid scope, standard login, logout, and token commands. The tests verify successful execution and appropriate responses for different user authentication scenarios, thereby increasing the robustness and reliability of the auth module.
* Update unit tests for auth commands
Unit tests for DVC Studio authentication commands have been updated. These checks verify successful execution and appropriate responses under various authentication scenarios, including login with invalid scope, standard login, logout, and token commands in order to enhance the robustness and reliability of the auth module.
* Add unit tests and modify authentication function in studio utils
This commit adds several new unit tests in test_studio.py. Tests for start_device_login and check_token_authorization functionality are included to verify the correct behavior of these methods. Moreover, it also modifies the authentication function in studio.py for improving the robustness. Changes in the function aimed to handle unexpected error scenarios and make the code more resilient.
* Refactor mock_response usage in tests and add test_auth.py
This commit refactors the way mock responses are used in test_studio.py to follow DRY principles, changing the private _mock_response to a public reusable function. It also adds a new file, test_auth.py, that provides functional tests for authorisation methods, ensuring their functionality. The patches in test_auth.py helps test different cases like expired authorization and successful authorization. Furthermore, test_auth.py also tests if correct configuration is set on successful authorization. Modifications on the method test_auth_login_arguments in test_auth.py were made to refine test cases, thus making the tests more precise and straightforward.
* Refactor error handling in `dvc/utils/studio.py`
Refactored error handling by removing redundant code blocks and introducing more detailed logging for request exceptions. This makes the failure cases handled explicitly and streamlines the traceability. Also, tests for authorization have been refined and more specific cases like 'expired authorization' have been included.
* Replace `auth` module with `studio` in DVC
Renamed all instances of `auth` to `studio`, as we are moving the authentication process into a new module `studio`. This includes changes to command names, classes, and test file names to reflect the updated module. It improves the organization of the authentication features related to DVC Studio, separating them from other types of authorization.
* Remove device login methods in utils/studio
Removed methods related to device login and token authorization in `utils/studio.py` and its corresponding test cases. These functionalities have been moved to the `dvc-studio-client` module to decouple the login process and make the code maintainable. Updated the dependencies in `pyproject.toml` to use the new `dvc-studio-client` module. Also, updated the command/studio to use the `dvc-studio-client` module for auth functionalities.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update dvc-studio-client dependency in pyproject.toml
The "dvc-studio-client" dependency in pyproject.toml is updated from a specific git commit to v0.16.0, removing the TODO comment. This change was made because the necessary features from the PR linked in the TODO comment have been merged and released in v0.16.0, making the git commit unnecessary. Now the code is cleaner and easier to track with versioned dependency.
* Update DVC Studio components and simplify authentication process
This commit modifies several parts of the ‘dvc/commands/studio.py’ and updates DVC Studio Client's version in 'pyproject.toml' for compatibility with the changes.
Primarily, the authentication process has been simplified. The unnecessary generation of a random token name is removed while initiating authorization in the run method of CmdStudioLogin class, enhancing the code quality.
Changes made in 'dvc/commands/studio.py' are mainly restructuring the code to adapt to the newer version of DVC Studio Client, including changes to the scopes and methods used from this client.
The corresponding changes were also reflected in unit tests. The test files 'tests/func/test_studio.py' and 'tests/unit/command/test_studio.py' are updated in accordance with the approach now used in 'dvc/commands/studio.py'.
* Remove unnecessary tests and update StudioClient methods
Removed functional test file `test_studio.py`from DVC to move it to studio client. The new mock response no longer needs a full testing file, simplifying our test environment.
In `dvc/commands/studio.py`, adjusted function imports to align with new StudioClient function names, which better indicate their purpose: `initiate_authorization` changed to `get_access_token`.
Additionally, `DEFAULT_SCOPES` was removed from both `dvc/commands/studio.py` and `tests/unit/command/test_studio.py`, simplifying the token acquisition process and removing unnecessary global variable. The `--scopes` command, if needed, can be passed directly without requiring a default fallback.
The corresponding changes were reflected in unit tests : the mock functions in `tests/unit/command/test_studio.py` were changed to align with the new authentication method, and updated the usage of the `get_access_token` function.
* Update token success message in dvc/commands/studio.py
This commit replaces f-string (formatted string literal) with a regular string for the success message after token authentication. The change was necessary as the variable `token_name` is already being passed as a formatted string, thus no need for f-string in the success message sentence. This makes code cleaner and more consistent.
* Refactor tests in `test_studio.py` for better readability
* Updated terminology in `studio.py` and `test_studio.py`
Changes terminology from 'authorize' to 'authenticate' to better align with common practices and improve understanding. Also fixed the `get_access_token` function in 'test_studio.py' where 'AuthorizationExpired' was replaced with 'AuthenticationExpired' for better error handling.
* Change 'dvc' to 'DVC' in `studio.py`
Updates all instances of 'dvc' to 'DVC' in `studio.py` for enhanced consistency with the rest of the project. This change aims to standardize the usage of the term across all project files.
* Improve Studio commands information clarity
Updated descriptions in various commands on 'dvc/commands/studio.py' focused on improving clarity and context. Authentication message, command helper, and descriptions have been expanded to provide better understanding. Also, 'dvc' has been changed to 'DVC' in 'studio.py' for consistency with the rest of the project files.
* Not relevant anymore
* Change argument name in studio login
Renamed the argument from "--use-device-code" to "--no-open" in the login function under the Studio authentication. The option's name was changed to make it more intuitive and better align with its actual effect, avoiding potential user confusion. Corresponding changes have also been made in unit tests.
* Modify flag for Studio login function
Updated the argument name from "-d" to "-o" in `studio.py` for the Studio login function. The change makes the argument's intention clearer for users. This is significant in improving user interaction with the feature and mitigating potential user misinterpretation.
* Apply suggestions from code review
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: skshetry <[email protected]>
* experimental: support connection strings for databases using connectorx
This adds an experimental support for connector strings using connectorx.
User has to provide config for the connection strings with:
Example:
```console
dvc config --local db_connection.pg postgres://user:pass@localhost:5432/database
```
And then, they can import using:
```console
dvc import-db --sql 'query' --conn pg
```
Check supported databases and how to configure in https://sfu-db.github.io/connector-x/databases.html.
This `db_connection` config is subject to change.
connectorx still has the same problem with dbt as before: no support for streaming,
but it'll be a bit more simpler to configure with the connection strings.
* add pandas to deps
* sqlalchemy
* dbt streaming sql
* fix test
* remove connectorx
* remove check in set
* use mode='wb' for serialization
skshetry
AlexandreKempf
pre-commit-ci[bot]
Matt Seddon
Matt Seddon
Dave Berenbaum
Ruslan Kuprieievanddberenbaum
Peter Rowlands
Ivan Shcheklein