Avoid raising errors in `data_status()` when Git HEAD does not contain a DVC
repository. Instead, fall back to an empty index.
Tests now cover both unborn and committed Git repos, with and without subdirs.
* Prioritize DVC_SITE_CACHE_DIR over core.site_cache_dir
* Add tests for proper prioritization of env var and config
* Tests require python-3.9 compatible typing, use Optional
* Remove comment
* Fixup
* DVC_SITE_CACHE_DIR must be sent first, deal with platformdirs afterwards
* Test should check default behaviour when no env var is set
* Add test: site_cache_dir() must return the env var if set
Clear logs before assertions so that logs from other tests do not interfere with the test.
Some destructors may also run in between due to GC, so hopefully this will fix that issue too,
or reduce the chances of it happening.
See
https://github.com/iterative/dvc/actions/runs/14873217814/job/41775314725#step:6:97.
```console
> assert first(caplog.messages) == expected_message
E assert 'RC: discarding all clients' == "Output 'path... 'stage.dvc'."
E
E + RC: discarding all clients
E - Output 'path'(stage: 'stage.dvc') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
E - You can also use `dvc commit stage.dvc` to associate existing 'path' with stage: 'stage.dvc'.
```
This `_get_remote_config()` should use `uninitialized=True` so that it supports
more broader kinds of broken/partially-initialized repositories, including some that have
`.dvc` directory missing, or `.git` directory missing.
This partially reverts #10719. #10608 is also fixed, and no longer requires `core.no_scm` to be passed.
This was already supported by `dvc.api.get_url` as it uses `uninitialized=True`, but this was not respected
in `_get_remote_config()` where it would fail before.
That said, this whole `open_repo`/`_get_remote_config` is terribly broken. For one, it is opening
a local repository, and forcing it's remote config to a repository opened with `Repo(rev=...)`,
where the config may be different.
* Forward args to _get_remote_config and honour core/no_scm if present
* Pass full config, except cache, to Repo()
* Avoid modifying original kwargs contents
* Simplify if config
* Add test
* Relax error matching.. .. Avoid windows vs unix paths
* Update dvc/repo/open_repo.py
---------
Co-authored-by: skshetry <[email protected]>
* remote: show default when run "dvc remote list"
Fixes#10708
* Indent other remotes if there is a default remote
Now regardless of whether there is a default remote or not, all remotes are equally indented
* stricter test for listing remotes shows default
* Rewrite dvc remote list implemetation
- Default remote is indicated at end of the output and colored green
- Remotes no longer reordered to show default first
- Test updated to reflect changes and remove need for function escape_ansi.
---------
Co-authored-by: RMCrean <[email protected]>
python/cpython#113829 changed `os.path.isabs` on Windows to not consider a path starting a single backslash to be an absolute path.
We do a lot of naive `posixpath` to `ntpath` conversions, and it broke DVC in multiple places. DVC is likely broken in many other places, and these bugs are difficult to identify.
In this commit, I have tried to fix in the places where the tests failed. And mostly by trying to imitate pre-3.13 behaviour.
Some of the changes may not be correct but keeps the old behaviour in place.
Also re-enabled the CI for all Python versions supported for Windows.
* fixed a bug where using both `--queue` and `-A` would yield a "removed" list without the queued experiments
* added a test to validate the fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* clearer name for test
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add keep_selected parameter, and corresponding code to keep only the selected exps (and remove all the other ones)
* test keep_selected_by_name
* test keep_selected_by_rev
* test keep_selected multiple, by name
* test keep all by name
* test keep by rev, with num=2
* added option to cli
* refactoring to meet pr needs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fixed test_experiments to add keep_selected=False to remove tests
* rename parameter to match cli option
* follow the normal path, then invert the selection before removing
* fixed tests for list ordering + fixed test with non existent name, it didn't make sense to delete everything if an exp name did not exist
* changed cli option comment
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fixed typing issue
* updated parameter name
* removed handling queued experiments (since --queue would remove them all)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* code simplification, added __eq__ and __hash__ to be able to compare ExpRefs, updated and parametrized tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fixed linting issues
* - --keep and --queue together raise an InvalidArgumentError
- added a test to check if the error is raised
- fixed CLI message
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* re-run gh tests. Some tests which did not involve my changes started failing while they were passing fine before.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* save import hash info to state
* clean up path handling
* avoid walking
* tests: do not use as_posix to check for output
---------
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
- When checkouting non-existent file the _checkout_file recursively
calls os.stat on a symlinked file which destination doesn't exist.
This causes a FileNotFoundError but it is not catched and it causes
an unexpected error instead.
Fixes#10500.
params/metrics: assume file when dir expansion raises error
Closes https://github.com/iterative/dvc/issues/10018.
We have to be careful when expanding paths. This PR fall backs to assume it
as a file when either of `isdir` and `find` raises error due to some mishaps.
The consequence of this assumption happens when we try to read the file.
In that case, if the directory exists, and we happen to try to open it
as a file, dvc will raise `IsADirectoryError`, but that is handled correctly
and won't fail catastrophically like this did.
`dvc add` command incorrectly raises a `DoesNotExistError` when a
broken symlink exists in an output directory, and the target name
is same as the directory's name.
eg: If `data` is an output, and is the command is invoked as `dvc add data`
(i.e. no virtual directory operations to perform).
The expected behavior to raise a `FileNotFoundError`.
`DoesNotExistError` should only be raised if the output itself does not
exist.
Related: https://github.com/iterative/dvc/issues/3717
* studio login set auto_push + add logs for the user
* set a separate global config dir for studio login tests
The `isolate` fixture sets a separate, _isolated_ global config directory.
But it is shared among other tests. So any mutations to it affects other
tests too.
We should force `studio login` tests to use a separate tmp_dir for global config.
* cleanup message
---------
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
* add auto_push for experiment on run and on save.
add config.exp.auto_push and config.exp.git_remote
* add func tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* separate auto_push behaviour out of commit
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
* Revert "limit pytest version to <8.1 (#10334)"
This reverts commit bc95b650a4.
* replace flaky with pytest-rerunfailures
`flaky` project seems to be unmaintained (see box/flaky#192).
`pytest-rerunfailures` seems to be maintained under `pytest-dev`
and tests 5 minor releases of pytest.
Note that this plugin is not compatible with `flaky` plugin.
You will have to uninstall it if it exists.
* ds add: use --dvc/--dvcx/--url flags to accept url
.. instead of relying on `scheme://`
I think this is more simpler from user's perspective, easier to
teach and easier to maintain as we won't have to do url parsing.
I am taking inspiration from `cargo install` has `--git`/`--rev`, etc.
to install from alternate sources.
```console
Options:
--version <VERSION> Specify a version to install
--index <INDEX> Registry index to install from
--registry <REGISTRY> Registry to use
--git <URL> Git URL to install the specified crate from
--branch <BRANCH> Branch to use when installing from git
--tag <TAG> Tag to use when installing from git
--rev <SHA> Specific commit to use when installing from git
--path <PATH> Filesystem path to local crate to install
--root <DIR> Directory to install packages into```
```
The `name` is also changed from an option to an argument.
### Examples
```console
dvc ds add example-get-started --dvc [email protected]:iterative/example-get-started.git
dvc ds add dogs --dvcx dogs
dvc ds add dogs --url s3://bucket/key/path
```
* add --version alias
* add support for tracking remote dataset
This adds support for virtually tracking a
dataset such as dvcx dataset, dataset from remote dvc/git
registries, and cloud-versioned remotes.
This PR introduces two different commands under `ds` namespace:
`add` and `update`.
`dvc ds add` command adds the dataset to `dvc.yaml` and tracks
the sources in `dvc.lock` file. Similarly, `dvc ds update`
updates the sources in `dvc.lock` file.
Example Usage:
```console
dvc ds add --name dogs --url dvcx://dogs # tracking dvcx dataset
# tracking dvc/git registries
dvc ds add --name example --url dvc://[email protected]/iterative/example.git
# cloud-versioned-remotes
dvc ds add --name versioning --url s3://cloud-versioning-demo
```
To update, specify the name.
```console
dvc ds update <name>
```
`dvc ds add` freezes (in the traditional sense of `dvc import/import-url`).
It keeps a "specification" of the dataset in `dvc.yaml` and also freezes
information about the dataset in the `dvc.lock` file. They are kept
inside `datasets` section in both `dvc.yaml` and `dvc.lock` files.
This metadata is used in the pipelines. You can add a dependency to your
stage using `ds://` scheme, followed by the name of the dataset in
`dvc.yaml` file. As it is used in pipelines, the `name` of the dataset
has to be unique in the repository. Different dataset of same names
are not allowed. On `dvc repro`, `dvc` copies the frozen information
about the particular dataset into the `deps` field for the stage in
`dvc.lock`. On successive invocation, `dvc` will compare the information
from the deps field of the lock with the frozen information in the
`datasets` section and decides whether to rerun or not.
As the dataset is frozen, `dvc repro` won't rerun until the dataset is
updated via `dvc update`.
Here are some examples for how `dvc.yaml` looks like:
```yaml
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
- name: dogs2
url: dvcx://dogs@v2
type: dvcx
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
```
```yaml
schema: '2.0'
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
rev_lock: df75c16ef61f0772d6e4bb27ba4617b06b4b5398
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
meta:
isdir: true
size: 323919
nfiles: 33
files:
- relpath: myproject/model.pt
meta:
size: 106433
version_id: 5qrtnhnQ4fBzV73kqqK6pMGhTOzd_IPr
etag: 3bc0028677ce6fb65bec8090c248b002
# truncated
```
The pipeline stages keep them in `dataset` section of individual deps.
```console
dvc stage add -n train -d ds://dogs python train.py
cat dvc.yaml
```
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- ds://dogs
```
When it is reproduced, the `dvc.lock` will look something like follows:
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- path: ds://dogs
dataset:
name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
```
* remodeling
* remove non-cloud-versioned remotes
* improve deserializing; handle invalidation
* dvc.api for dataset
* expose the API in the top-level
* add support for tracking remote dataset
This adds support for virtually tracking a
dataset such as dvcx dataset, dataset from remote dvc/git
registries, and cloud-versioned remotes.
This PR introduces two different commands under `ds` namespace:
`add` and `update`.
`dvc ds add` command adds the dataset to `dvc.yaml` and tracks
the sources in `dvc.lock` file. Similarly, `dvc ds update`
updates the sources in `dvc.lock` file.
Example Usage:
```console
dvc ds add --name dogs --url dvcx://dogs # tracking dvcx dataset
# tracking dvc/git registries
dvc ds add --name example --url dvc://[email protected]/iterative/example.git
# cloud-versioned-remotes
dvc ds add --name versioning --url s3://cloud-versioning-demo
```
To update, specify the name.
```console
dvc ds update <name>
```
`dvc ds add` freezes (in the traditional sense of `dvc import/import-url`).
It keeps a "specification" of the dataset in `dvc.yaml` and also freezes
information about the dataset in the `dvc.lock` file. They are kept
inside `datasets` section in both `dvc.yaml` and `dvc.lock` files.
This metadata is used in the pipelines. You can add a dependency to your
stage using `ds://` scheme, followed by the name of the dataset in
`dvc.yaml` file. As it is used in pipelines, the `name` of the dataset
has to be unique in the repository. Different dataset of same names
are not allowed. On `dvc repro`, `dvc` copies the frozen information
about the particular dataset into the `deps` field for the stage in
`dvc.lock`. On successive invocation, `dvc` will compare the information
from the deps field of the lock with the frozen information in the
`datasets` section and decides whether to rerun or not.
As the dataset is frozen, `dvc repro` won't rerun until the dataset is
updated via `dvc update`.
Here are some examples for how `dvc.yaml` looks like:
```yaml
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
- name: dogs2
url: dvcx://dogs@v2
type: dvcx
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
```
```yaml
schema: '2.0'
datasets:
- name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
- name: example-get-started
url: [email protected]:iterative/example-get-started.git
type: dvc
path: path
rev_lock: df75c16ef61f0772d6e4bb27ba4617b06b4b5398
- name: cloud-versioning-demo
url: s3://cloud-versioning-demo
type: url
meta:
isdir: true
size: 323919
nfiles: 33
files:
- relpath: myproject/model.pt
meta:
size: 106433
version_id: 5qrtnhnQ4fBzV73kqqK6pMGhTOzd_IPr
etag: 3bc0028677ce6fb65bec8090c248b002
# truncated
```
The pipeline stages keep them in `dataset` section of individual deps.
```console
dvc stage add -n train -d ds://dogs python train.py
cat dvc.yaml
```
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- ds://dogs
```
When it is reproduced, the `dvc.lock` will look something like follows:
```yaml
# truncated
stages:
train:
cmd: python train.py
deps:
- path: ds://dogs
dataset:
name: dogs
url: dvcx://dogs
type: dvcx
version: 3
created_at: '2023-12-11T10:32:05.942708+00:00'
```
* remodeling
* remove non-cloud-versioned remotes
* improve deserializing; handle invalidation
This was introduced in Python 3.9. The change makes
dict merges 'obvious' and looks cleaner.
I haven't changed `.update()` to `|=` (update operator),
as I find the former to be more readable and "obvious".
* repo: site_cache_dir: separate subdir from root_dir
Currently we join scm root_dir with subrepo root dir to have something that would represent that subrepo, but that directory might not exist in reality as we might be using a bare git repo. So we should just treat those as separate things.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update __init__.py
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The code are wrapped due to us using line-length=79 before, but now some
of the code can easily fit in 88 chars.
The formatter did not change this automatically when we changed line-length,
because of magic trailing commas that gets added when they turn into multiple lines.
Black/ruff wrapping them in parens and splitting them into multiple lines
was making it difficult to read.
* drop Python 3.8
Note that this PR only makes typing related changes.
Other changes can be done once this gets merged, they
are relatively few in numbers.
* bump build python version
* add typing_extensions to lint group
* remote: add user key to config schema webhdfs
This was already supported according to documentation, but missing in
code
Fixes: 10062
* remote: add password key to support Basic Auth
* remote: add data_proxy_target for webhdfs remote
This key is needed to construct a potential rewrite dictionary in
webhdfs to deal with HDFS behind High Availability Proxy servers
* bump dvc-webhdfs
---------
Co-authored-by: skshetry <[email protected]>
* param: simplify fix for exponential memory allocation
* Apply suggestions from code review
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove `deepcopy` again
* Restore deepcopy
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* param: fix exponential memory allocation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sometimes seeing those error out in 2.x and early 3.x with:
```
ERROR: unexpected error - [Errno 95] no more link types left to try out: [Errno 95] Operation not supported
```
It seems like we've fixed it at some point, but I was not able to quickly find a link.
* standardise plot data output
* add data required for new anchors to renderer properties
* feed dvc-render updates into API output
* update --json --split api with new data types (send split anchor definitions as strings)
* accomodate zoom and pan anchor
* switch anchor_revs to revs_with_datapoints
* accomodate height and width anchors
* drop terrible idea of holding all data as strings
* fix plot having multiple x fields
* move inferred properties off vega converter class
* Add Studio authentication to DVC
Implemented a new feature to authenticate DVC with Iterative Studio. This includes generating an access token, starting device login, checking if user code is authorized, and handling any request exceptions. Furthermore, the commit creates a new authentication command for DVC. This feature was added to allow DVC users to authenticate with Studio, which is necessary for functionality such as sharing live experiments and notifying Studio about pushed experiments.
* Add authentication validation and error handling to DVC Studio
This commit introduces validation for scopes used in DVC Studio's authentication process. If an unsupported scope is passed, the user will receive an error message listing the invalid scopes and the program will return an error status. Additionally, this commit changes the program's exit behavior when device code authorization fails or authentication process is successful, replacing 'sys.exit' with appropriate returned values.
* Add logout functionality to DVC Studio commands
This commit introduces the logout command for DVC Studio. It allows users to manually log out from DVC Studio via the command line interface. The logout command only removes the user's token if it exists, otherwise, it will print an error message and return an error status. This adds an extra layer of control for the users and enhances the security for the usage of DVC Studio.
* Improve User Authentication process for DVC Studio
Implemented a new 'token' command facilitating users to check their authentication token while operating with DVC Studio. This implementation enhances user experience by providing the ability to validate logged-in status. If not logged in, it rightly prompts an error, ensuring secure access.
* Add unit tests for auth commands
Unit tests are added for the authentication commands used in DVC Studio. The modifications include tests for login with invalid scope, standard login, logout, and token commands. The tests verify successful execution and appropriate responses for different user authentication scenarios, thereby increasing the robustness and reliability of the auth module.
* Update unit tests for auth commands
Unit tests for DVC Studio authentication commands have been updated. These checks verify successful execution and appropriate responses under various authentication scenarios, including login with invalid scope, standard login, logout, and token commands in order to enhance the robustness and reliability of the auth module.
* Add unit tests and modify authentication function in studio utils
This commit adds several new unit tests in test_studio.py. Tests for start_device_login and check_token_authorization functionality are included to verify the correct behavior of these methods. Moreover, it also modifies the authentication function in studio.py for improving the robustness. Changes in the function aimed to handle unexpected error scenarios and make the code more resilient.
* Refactor mock_response usage in tests and add test_auth.py
This commit refactors the way mock responses are used in test_studio.py to follow DRY principles, changing the private _mock_response to a public reusable function. It also adds a new file, test_auth.py, that provides functional tests for authorisation methods, ensuring their functionality. The patches in test_auth.py helps test different cases like expired authorization and successful authorization. Furthermore, test_auth.py also tests if correct configuration is set on successful authorization. Modifications on the method test_auth_login_arguments in test_auth.py were made to refine test cases, thus making the tests more precise and straightforward.
* Refactor error handling in `dvc/utils/studio.py`
Refactored error handling by removing redundant code blocks and introducing more detailed logging for request exceptions. This makes the failure cases handled explicitly and streamlines the traceability. Also, tests for authorization have been refined and more specific cases like 'expired authorization' have been included.
* Replace `auth` module with `studio` in DVC
Renamed all instances of `auth` to `studio`, as we are moving the authentication process into a new module `studio`. This includes changes to command names, classes, and test file names to reflect the updated module. It improves the organization of the authentication features related to DVC Studio, separating them from other types of authorization.
* Remove device login methods in utils/studio
Removed methods related to device login and token authorization in `utils/studio.py` and its corresponding test cases. These functionalities have been moved to the `dvc-studio-client` module to decouple the login process and make the code maintainable. Updated the dependencies in `pyproject.toml` to use the new `dvc-studio-client` module. Also, updated the command/studio to use the `dvc-studio-client` module for auth functionalities.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update dvc-studio-client dependency in pyproject.toml
The "dvc-studio-client" dependency in pyproject.toml is updated from a specific git commit to v0.16.0, removing the TODO comment. This change was made because the necessary features from the PR linked in the TODO comment have been merged and released in v0.16.0, making the git commit unnecessary. Now the code is cleaner and easier to track with versioned dependency.
* Update DVC Studio components and simplify authentication process
This commit modifies several parts of the ‘dvc/commands/studio.py’ and updates DVC Studio Client's version in 'pyproject.toml' for compatibility with the changes.
Primarily, the authentication process has been simplified. The unnecessary generation of a random token name is removed while initiating authorization in the run method of CmdStudioLogin class, enhancing the code quality.
Changes made in 'dvc/commands/studio.py' are mainly restructuring the code to adapt to the newer version of DVC Studio Client, including changes to the scopes and methods used from this client.
The corresponding changes were also reflected in unit tests. The test files 'tests/func/test_studio.py' and 'tests/unit/command/test_studio.py' are updated in accordance with the approach now used in 'dvc/commands/studio.py'.
* Remove unnecessary tests and update StudioClient methods
Removed functional test file `test_studio.py`from DVC to move it to studio client. The new mock response no longer needs a full testing file, simplifying our test environment.
In `dvc/commands/studio.py`, adjusted function imports to align with new StudioClient function names, which better indicate their purpose: `initiate_authorization` changed to `get_access_token`.
Additionally, `DEFAULT_SCOPES` was removed from both `dvc/commands/studio.py` and `tests/unit/command/test_studio.py`, simplifying the token acquisition process and removing unnecessary global variable. The `--scopes` command, if needed, can be passed directly without requiring a default fallback.
The corresponding changes were reflected in unit tests : the mock functions in `tests/unit/command/test_studio.py` were changed to align with the new authentication method, and updated the usage of the `get_access_token` function.
* Update token success message in dvc/commands/studio.py
This commit replaces f-string (formatted string literal) with a regular string for the success message after token authentication. The change was necessary as the variable `token_name` is already being passed as a formatted string, thus no need for f-string in the success message sentence. This makes code cleaner and more consistent.
* Refactor tests in `test_studio.py` for better readability
* Updated terminology in `studio.py` and `test_studio.py`
Changes terminology from 'authorize' to 'authenticate' to better align with common practices and improve understanding. Also fixed the `get_access_token` function in 'test_studio.py' where 'AuthorizationExpired' was replaced with 'AuthenticationExpired' for better error handling.
* Change 'dvc' to 'DVC' in `studio.py`
Updates all instances of 'dvc' to 'DVC' in `studio.py` for enhanced consistency with the rest of the project. This change aims to standardize the usage of the term across all project files.
* Improve Studio commands information clarity
Updated descriptions in various commands on 'dvc/commands/studio.py' focused on improving clarity and context. Authentication message, command helper, and descriptions have been expanded to provide better understanding. Also, 'dvc' has been changed to 'DVC' in 'studio.py' for consistency with the rest of the project files.
* Not relevant anymore
* Change argument name in studio login
Renamed the argument from "--use-device-code" to "--no-open" in the login function under the Studio authentication. The option's name was changed to make it more intuitive and better align with its actual effect, avoiding potential user confusion. Corresponding changes have also been made in unit tests.
* Modify flag for Studio login function
Updated the argument name from "-d" to "-o" in `studio.py` for the Studio login function. The change makes the argument's intention clearer for users. This is significant in improving user interaction with the feature and mitigating potential user misinterpretation.
* Apply suggestions from code review
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: skshetry <[email protected]>
* testing: bench: add pytest_report_header
Useful for sanity checks in logs.
* Update dvc/testing/benchmarks/plugin.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* experimental: support connection strings for databases using connectorx
This adds an experimental support for connector strings using connectorx.
User has to provide config for the connection strings with:
Example:
```console
dvc config --local db_connection.pg postgres://user:pass@localhost:5432/database
```
And then, they can import using:
```console
dvc import-db --sql 'query' --conn pg
```
Check supported databases and how to configure in https://sfu-db.github.io/connector-x/databases.html.
This `db_connection` config is subject to change.
connectorx still has the same problem with dbt as before: no support for streaming,
but it'll be a bit more simpler to configure with the connection strings.
* add pandas to deps
* sqlalchemy
* dbt streaming sql
* fix test
* remove connectorx
* remove check in set
* use mode='wb' for serialization
We should not register a pytest plugin automaticallly, as we may
have fixtures that might override user's fixtures. Also, it might
raise import errors as in https://github.com/iterative/dvc/issues/10118.
We can still use this as a plugin ourselves by setting the following
in `conftest.py` file.
```python
pytest_plugins = ['dvc.testing.plugin']
```
Previously we used recorded index which had stub entries
for missing/failed outputs, but now we actually collect
everything ourselves and those are missing.
* remove pytest-virtualenv
It does not look like it's being actively maintained.
* force virtualenv
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
__name__ looks something like `dvc.commands.diff`, but if we are already instantiating
root logger with `dvc`, it adds up to create something like `dvc.dvc.commands.diff`.
We should be using a root logger here instead.
* use logger.getChild from `dvc.log` instead of `logging.getLogger`
I got tired adding `# type: ignore[attr-defined]` while using `logger.trace`.
So, now you need to import logger from `from dvc.log import logger` and
create a child logger using `logger.getChild(__name__)`. `logger.getChild`
will return a logger type-hinted as a logger having a `trace` method.
So, from now on, instead of:
```python
import logging
logger = logging.getogger(__name__)
```
you'll have to do following:
```python
from dvc.log import logger
logger = logger.getChild(__name__)
```
* remove all dvc imports from dvc.logger
* use Tqdm back
This commit adds the `key` property to the templaing context of `matrix`
stages. Similarly to `foreach` stages, the `key` is available using
`${key}` substitution.
Fixes#10043
See https://docs.python.org/3/library/sys.html#sys.__stdin__.
> [!Note]
> Under some conditions stdin, stdout and stderr as well as the original values __stdin__, __stdout__ and __stderr__ can be None. It is usually the case for Windows GUI apps that aren’t connected to a console and Python apps started with pythonw.
This is a best-effort solution. External libraries (eg: `tqdm`) and future code that is unaware of this stuff may still fail.
As a console application, `stdout`/`stderr` being `None` is not really something that we should support in general.
Regarding the failure of the daemon, it has been fixed so that it'll always have a stream attached (if not, it falls back to
/dev/null). Today, this is a problem in daemon only if the main process is run with no streams attached, which is not
something that we support or go on length to support.
It is tricky to write tests for this case, and I could not come up with one. Also decided not to test daemon as it should no longer be a problem.
* use popen to run updater/analytics in detached mode
subprocess does not provide a way to avoid a zombie process. Not to confuse with orphaned processes, which will be `wait`-ed by `pid 1`.
It might not work for long-running processes, but I don't think zombie processes are going to be a problem for us, as the parent process immediately exits after spawning a child process.
Even in case of `queue` based experiment runner, parent will exit before than the child process. So I don't think it's a requirement here. But it does feel wrong to leak the processes, so I have to sleep on that.
---
This PR:
1. makes `daemonized` process inherit the logging verbosity, so if the parent process used `-vv`, the daemonized process will run the same way.
2. Add `DVC_DAEMON_LOGFILE` envvar, that the daemon will log to the file. By default, the verbosity is the same as running parent processes. Without `DVC_DAEMON_LOGFILE`, the output will be redirected to `/dev/null`.
3. Adds debug/trace for updater/analytics.
I hope by doing this, we can make our life easier on debugging and make daemon more transparent/observable.
* add tests
* fixes
* use kwargs
* pass all os.environ with some modified
* artifacts: Bring back `name_is_compatible`.
Unfortunately, DVCLive was importing this function so I am bringing back the name for backwards compatibility.
* Update artifacts.py
There is more we can do during the `Fetching` stage, but it will
require additional tweaks, so just keeping it simple here.
Similar to https://github.com/iterative/dvc/pull/9974
Existing logic might result in weird fs cache location like
`.dvc/cache/files/md5/fs`, while we really only want
`.dvc/cache/fs`, since it is not an odb cache.
If user uses some custom cache location, there is really no reason for us
to try to gitignore it, as we have no idea what he is planning to do with
that and overall it often might be sign of this being misused. Maybe we could
also have some warnings/errors, but that's out of scope for this PR.
Fixes https://github.com/iterative/studio/issues/7405
* exp show: show metrics that include separator
* sort using partition instead of lpartition
* handle ':' in file and metric names
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix sorting with leading :
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* update msg for multiple overlapping outputs
* update msg for multiple overlapping outputs
* fix type checking
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use sets of stages to fix mypy errors
* fix tests/func/test_add.py::test_try_adding_multiple_overlaps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] pre-commit autoupdate
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.0.282 → v0.0.285](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.282...v0.0.285)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix ruff issues
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
Currently we get odd formatting like
```
456 .devcontainer.json
139 .dvcignore
61.0 .gitattributes
96.0 .github
7.00 .gitignore
5.76k README.md
160 data
1.84k dvc.lock
519 dvc.yaml
4.00 foo
96.0 models
96.0 notebooks
225 params.yaml
46.0 requirements.txt
128 results
160 src
```
(notice stuff like 46.0)
and there doesn't seem to be a simple way to fix that in tqdm, in addition to
the general feeling that we are abusing tqdm's util for this.
So let's just use `humanize`, which gets us a more pleasing result:
```
456B .devcontainer.json
139B .dvcignore
61B .gitattributes
96B .github
7B .gitignore
5.8K README.md
160B data
1.8K dvc.lock
519B dvc.yaml
4B foo
96B models
96B notebooks
225B params.yaml
46B requirements.txt
128B results
160B src
```
Likely with removal of dvc-machine pyinstaller stopped picking up dvc_ssh automatically, so we need to declare it explicitly just like the rest of plugins.
* use _describe in exp list
* fix pre-commit checks
* simplistic fix
* single _describe, too complex
* slightly simpler
* even simpler
* simple, but with regex
* less substitutions
* describe public
* name handling in UI (tests need updating)
* fixed old tests
* lazy imports
---------
Co-authored-by: Tibor Mach <[email protected]>
This used to work like so but by accident, when we didn't reload config
at all when collecting other revisions.
Regression from https://github.com/iterative/dvc/pull/9758
* hydra: enable config modules
Lets the user specify Hydra configs in installed Python modules as well
as inside directories. Adds `hydra.config_module` to `.dvc/config`.
Fixes#9740
old tests green
tests for config modules
* hydra: fix linter errors
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* hydra: fix mypy error
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* hydra: fix tests
* Update dvc/utils/hydra.py
Co-authored-by: David de la Iglesia Castro <[email protected]>
* address comments
* convince mypy not to worry
* split long line
* fix test
* check for raised error in test
---------
Co-authored-by: Dom Miketa <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David de la Iglesia Castro <[email protected]>
`dvc commit --help` was also completely broken, which is also fixed.
This adds unit tests for completion generation for both zsh/bash, and
tests that --help works for all commands.
Adds support for `--config` to all three commands. For `ls` and `get`, it is
a purely runtime option that will merge the config you provide with target
repo's configs (similar to how .dvc/config.local works). E.g.
```
$ cat myconfig
[remote "myremote"]
access_key_id = 12345
secret_access_key = 98765
$ dvc get https://github.com/efiop/mydataregistry recent-grads.csv --config myconfig
```
In case of `dvc import`, the semantics are the same, but the value is also
recorded in the resulting dvcfile. E.g.
```
$ dvc import https://github.com/efiop/mydataregistry recent-grads.csv --config myconfig
...
$ cat recent-grads.csv.dvc
md5: 85c86eeb3afa6d1f7d11beedc644d28e
frozen: true
deps:
- path: recent-grads.csv
repo:
url: https://github.com/efiop/mydataregistry
config: myconfig
rev_lock: af22e06bbfe34d62a140437abf32f48b535944a7
outs:
- md5: f447e23a170a9f25a7799f069a1ba934
size: 26872
hash: md5
path: recent-grads.csv
```
This is the most general and powerful mechanism that we can give, that works
for everything from reconfiguring your default remote to completely changing
your remote layout and beyond (will be handy in the future).
Fixes#2466
* repro: introduce --keep-going/--ignore to continue on error
`-k`/`--keep-going` will skip all dependents of a stage if it fails.
It'll still try to reproduce the other dependencies of the pending
targets, before it exits with a non-zero status.
`--ignore-errors` will treat any error during stage execution as a
success, and will continue executing other dependencies as if it
succeeded. Exits with 0 even on failure.
Also fixes:
* --force-downstream used to force everything even if one upstream nodes changed.
This used to work when we supported a single targets, but there could be independent
pipelines, so we should not force everything if any node changes.
Now we'll force reproduce the stage only if one of its dependency changed upstream
or it's upstream node was also forced.
* Makes `--downstream` more correct, in relation to `--allow-missing` and `--pull`.
Now we use an active graph for downstream/upstream logic instead of a subgraph.
Note that this does not affect the downstream logic though, as dvc always goes
downstream, so subgraph was no different than active graph.
The subgraph only did not have the upstream nodes.
* add tests
* add spacing in the message
* environment variable to show debug options
* lint
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add message to debug opts
* spelling
* Add argument group and move cprofile-dump
line length
linelen
lint
* use DVC_DEBUG instead of DVC_SHOW_DEBUG_OPTIONS
* use -v instead of environ
* Update dvc/_debug.py
Co-authored-by: skshetry <[email protected]>
* fix
---------
Co-authored-by: skshetry <[email protected]>
The logic is now moved to experiments, where it tracks all the stage related files
itself. The `_get_stage_files()` has been modified to also return lockfile path,
so that it can be staged, and has been moved to `stage/utils.py`, and out of `reproduce.py`.
Fixes#9688.
* repro: make stages run in order of the graph.
In case of --single-item, this means it'll run in the order of arguments passed.
For --downstream, this fixes running of stage out of order when multiple arguments are passed.
For normal case, this enforces the same logic, so that they are never run out-of-order.
* add tests
* tests: repro: add test for --single-item with multiple targets
* graph: support when no nodes are provided
Looks like flufl-lock switched to pdm in 8.0 which triggered a
```
pkg_resources.DistributionNotFound: The 'flufl.lock>=5' distribution was not found and is required by dvc
```
error in multiple scenarious. Can't put my finger on anything specific
in flufl-lock yet, so will just limit its version for now.
Fixes https://github.com/iterative/dvc/issues/9654
* [ ] ❗ I have followed the [Contributing to DVC](https://dvc.org/doc/user-guide/contributing/core) checklist.
* [ ] 📖 If this PR requires [documentation](https://dvc.org/doc) updates, I have created a separate PR (or issue, at least) in [dvc.org](https://github.com/iterative/dvc.org) and linked it here.
* [ ] 📖 If this PR requires [documentation](https://dvc.org/doc) updates, I have created a separate PR (or issue, at least) in [dvc.org](https://github.com/treeverse/dvc.org) and linked it here.
Thank you for the contribution - we'll try to review it as soon as possible. 🙏
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
pre-commit-ci[bot]
Staszek Markowski
Saugat Pachhai (सौगात)
dependabot[bot]
KoyoMatsuda-hacarus
B. John Braddock
Samandar MirjalilovandSaugat Pachhai
Italos Estilon, PhDandSaugat Pachhai
Thorvald Molthe BallestadandSaugat Pachhai
Rodrigo Goya
Clouds
Fabian Zills
Ivan Shcheklein
Rory CreanandRMCrean
Raphaël Michelandpre-commit-ci[bot]
Pete Bachant
Saipavan Lingamallu
Santiago SmithandSantiago Silva
Dave BerenbaumandSaugat Pachhai
Jan Stratil
Dave Berenbaum
George Geddes
George KussumotoandSaugat Pachhai
Sigurd Spieckermann
Thomas Roehr
Amirmohammad Sadat Shokouhi
Rishabh SharmaandRishabh Sharma
AlexandreKempfandSaugat Pachhai
Matt Seddon
Peter Rowlands
Ruslan Kuprieiev
Matt Seddon
Guido Diepenandskshetry
Piotr Bielak
Heinz-Alexander Fuetterer
Brady Johnson
David de la Iglesia Castro
Bilel Omrani
Joe Scott
Tibor MachandTibor Mach
Jon Crallandskshetry