Commit Graph
9081 Commits
Author SHA1 Message Date
skshetry 7c61463d2d tests: do not setup pytest plugin entrypoint (#10121)
We should not register a pytest plugin automaticallly, as we may
have fixtures that might override user's fixtures. Also, it might
raise import errors as in https://github.com/iterative/dvc/issues/10118.

We can still use this as a plugin ourselves by setting the following
in `conftest.py` file.
```python
pytest_plugins = ['dvc.testing.plugin']
```
2023-11-29 12:05:46 +05:45
Peter Rowlands b0d0315689 import/get: support pre-fetching LFS objects from Git-LFS repos 2023-11-29 14:39:54 +09:00
Peter Rowlands 7edcb6b1e5 deps: bump scmrepo minver to 1.5.0 2023-11-29 14:39:54 +09:00
Ruslan Kuprieiev b6d231f9cf testing: import virtualenv lazily
Fixes https://github.com/iterative/dvc/issues/10118
3.30.3
2023-11-28 22:20:34 +02:00
Ruslan Kuprieiev 322fcec5ca bench: test_sharing: use fetch instead of pull 2023-11-28 19:31:32 +02:00
Ruslan Kuprieiev 1e5b3603cc bench: test fetch instead of pull
`pull` is literally `fetch` + `checkout` and our main interest there is downloading, so let's just test fetch instead.
2023-11-28 19:31:32 +02:00
Ruslan Kuprieiev ffa630e8d5 push: handle missing/failed version_aware file
Previously we used recorded index which had stub entries
for missing/failed outputs, but now we actually collect
everything ourselves and those are missing.
3.30.2
2023-11-28 18:52:46 +02:00
Ruslan Kuprieiev 1e9b60dc48 deps: bump dvc-data to >=2.22.1 2023-11-27 16:17:42 +02:00
skshetry 83dd2b2a7b test: import-db: test for external model (#10112)
* test: import-db: test for external model

* close repo
2023-11-27 16:13:44 +05:45
Ivan Shcheklein d238350a2d fix(config): respect global config env variable 2023-11-23 20:51:52 +02:00
skshetry 06b9196371 pytest.ini_options: add --benchmark-skip to suppress pytest-benchmark warning with xdist (#10106)
We don't run benchmarks here, and benchmarking should always be a separate command than regular test run.
2023-11-23 19:55:23 +05:45
skshetry b4f42242c9 tmp_dir: fix object initialization for Python 3.12 (#10105) 2023-11-23 16:00:30 +05:45
skshetryandpre-commit-ci[bot] fbdb566079 remove pytest-virtualenv (#10103)
* remove pytest-virtualenv

It does not look like it's being actively maintained.

* force virtualenv

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-11-23 14:53:39 +05:45
skshetry 3d7bbb8492 tests: add functional test for db (#10101)
* tests: add functional test for db

* fix test
2023-11-22 16:15:07 +05:45
skshetry fd7c891217 implement import-db command (#10040)
* implement import-db command

* generate .dvc file

* update

* fix tests

* pin dbt-fal

* try to run on ci

* use os.chdir

* fix

* revert

* require dbt-core>=1.5

* add db config schema

* remove platformdirs constraints

* use logger.getChild

* improve error messaging

* rename cli flag --export-format to --output-format

* use feature flag for db config

* hide command from dvc commands list

* refactor dependency._get

* remove todo

* refactor dependency._get

* fix dbt installation check
2023-11-21 19:38:50 +05:45
Ruslan Kuprieiev 2b3b15f767 deps: bump dvc-data to 2.22.0
Related https://github.com/iterative/dvc/issues/10059
3.30.1
2023-11-17 20:01:57 +02:00
Ruslan Kuprieiev 8c9358957c push: fix cloud versioning 3.30.0 2023-11-15 16:28:17 +02:00
Ivan Shcheklein 86dd0785a0 add tests for the index param keys 2023-11-14 13:21:44 +02:00
Ivan Shcheklein a8aec51c35 add params collection to index 2023-11-14 13:21:44 +02:00
Ruslan Kuprieiev e9f2da4451 repo: drop_data_index: delete all first level nodes 3.29.0 2023-11-13 11:23:08 +02:00
dependabot[bot]andRuslan Kuprieiev 8d23a5e65a build(deps-dev): bump mypy from 1.6.1 to 1.7.0 (#10088)
* build(deps-dev): bump mypy from 1.6.1 to 1.7.0

Bumps [mypy](https://github.com/python/mypy) from 1.6.1 to 1.7.0.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v1.6.1...v1.7.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix misc mypy errors

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2023-11-13 10:59:32 +02:00
skshetry b2fc194e2b fix name of the loggers used (#10085)
__name__ looks something like `dvc.commands.diff`, but if we are already instantiating
root logger with `dvc`, it adds up to create something like `dvc.dvc.commands.diff`.

We should be using a root logger here instead.
2023-11-10 15:54:08 +05:45
dependabot[bot]andSaugat Pachhai 0def298b8c build(deps-dev): bump mypy from 1.5.1 to 1.6.1 (#10035)
* build(deps-dev): bump mypy from 1.5.1 to 1.6.1

Bumps [mypy](https://github.com/python/mypy) from 1.5.1 to 1.6.1.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v1.5.1...v1.6.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* pre-commit autoupdate

* mypy fixes

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-11-10 15:40:57 +05:45
skshetry d678773b53 remove more pylint suppression comments (#10084) 2023-11-10 15:19:52 +05:45
skshetry 519cce3fe4 logger: type-safe logger.trace (#10083)
* use logger.getChild from `dvc.log` instead of `logging.getLogger`

I got tired adding `# type: ignore[attr-defined]` while using `logger.trace`.
So, now you need to import logger from `from dvc.log import logger` and
create a child logger using `logger.getChild(__name__)`. `logger.getChild`
will return a logger type-hinted as a logger having a `trace` method.

So, from now on, instead of:
```python
import logging

logger = logging.getogger(__name__)
```

you'll have to do following:
```python
from dvc.log import logger

logger = logger.getChild(__name__)
```

* remove all dvc imports from dvc.logger

* use Tqdm back
2023-11-10 12:55:49 +05:45
skshetry cbc111b519 remove flake8/bandit (#10082)
ruff already runs most of the flake8/bandit checks
2023-11-09 21:26:37 +05:45
skshetry 4ee8f554a5 use ruff formatter (#10081)
Replaces black with ruff-formatter, and isort with ruff.
2023-11-09 20:58:58 +05:45
skshetry 06565e18b4 remove pylint suppression comments (#10080) 2023-11-09 20:31:50 +05:45
skshetry 2085bc8053 drop pylint from linter checks (#10079)
drop pylint
2023-11-09 19:25:28 +05:45
Peter Rowlands 14cbf6470d artifacts get: fix issue where global/system config could be ignored 2023-11-09 16:19:02 +09:00
Ivan Shcheklein 36fa116e75 chore(deps): bump scmerepo to 1.4.1+
After this fix https://github.com/iterative/scmrepo/issues/280, to make sure that we don't install by mistake an older version in Studio.
2023-11-06 15:23:18 +09:00
Ivan Shcheklein 97f3d07480 fix(readme): link to VS Code extension (#10063) 2023-11-02 20:37:32 -07:00
dberenbaum 4a2c4f76e5 fix for DVCFileSystem().ls() 2023-10-31 23:38:21 +02:00
dberenbaum eb8999a86b gc: force no prompts during dry run 2023-10-31 10:24:01 -05:00
dberenbaum db870335e3 gc: don't prompt when for dry run 2023-10-31 10:24:01 -05:00
Piotr Bielak 23d2ed3d59 Unroll dict-comprehension in matrix test 2023-10-31 08:05:38 -05:00
skshetry 46db3ac419 Update dvc/parsing/__init__.py 2023-10-31 08:05:38 -05:00
Piotr Bielak fcc69ab6eb parsing: Add key to matrix stages
This commit adds the `key` property to the templaing context of `matrix`
stages. Similarly to `foreach` stages, the `key` is available using
`${key}` substitution.

Fixes #10043
2023-10-31 08:05:38 -05:00
Ruslan Kuprieiev d60557dd08 deps: bump dvc-data to >=2.20.0 3.28.0 2023-10-31 04:51:30 +02:00
dberenbaum fc5147767f api: make exp_show return float instead of None for zeroes 2023-10-30 13:54:31 -05:00
skshetry 89b4beb7a4 daemon: ipc to log pid from the child (#10058)
* daemon: ipc to log pid from the child

* fix tests to use pid

* also test for updater/analytics keyword
3.27.1
2023-10-30 20:47:52 +05:45
skshetry 48f1f52c1b refactor: dvc.daemon (#10045)
* minor refactor to dvc.daemon

* fix tests

* run daemon only inside a subprocess
2023-10-30 11:09:55 +05:45
Ruslan Kuprieiev 3fee81f1be push: use index push 2023-10-26 20:58:45 +03:00
Peter Rowlands 0bfce0b643 exp apply: use force dvc-checkout by default 2023-10-26 14:48:41 +09:00
Peter Rowlands d9b2e67a0b Revert "Enable forcing exp apply (unsaved files in the workspace)"
This reverts commit 3661b8998b.
2023-10-26 14:48:41 +09:00
Ruslan Kuprieiev 4a0d56a793 cli: DictAction: set default metavar (#10047) 2023-10-20 16:46:14 +03:00
skshetry 825c823a37 avoid failing on standard streams that are None. (#10041)
See https://docs.python.org/3/library/sys.html#sys.__stdin__.

> [!Note]
> Under some conditions stdin, stdout and stderr as well as the original values __stdin__, __stdout__ and __stderr__ can be None. It is usually the case for Windows GUI apps that aren’t connected to a console and Python apps started with pythonw.

This is a best-effort solution. External libraries (eg: `tqdm`) and future code that is unaware of this stuff may still fail.
As a console application, `stdout`/`stderr` being `None` is not really something that we should support in general.
Regarding the failure of the daemon, it has been fixed so that it'll always have a stream attached (if not, it falls back to
/dev/null). Today, this is a problem in daemon only if the main process is run with no streams attached, which is not
something that we support or go on length to support.

It is tricky to write tests for this case, and I could not come up with one. Also decided not to test daemon as it should no longer be a problem.
2023-10-19 22:16:26 +05:45
skshetry b23ba58cd6 tests: daemon: log body of the request (#10039)
log body of the request
2023-10-19 17:16:34 +05:45
skshetry 8fb67b71a5 expose test webserver as an executable for easier testing (#10037) 2023-10-19 16:28:47 +05:45
skshetry 5693775459 refactor: use int as the factory (#10036)
makes it thread-safe if we need in the future. Also looks a bit clean.
2023-10-19 15:31:33 +05:45