Commit Graph
7219 Commits
Author SHA1 Message Date
dependabot[bot] a3dece8236 build(deps-dev): Bump types-tabulate from 0.8.4 to 0.8.5
Bumps [types-tabulate](https://github.com/python/typeshed) from 0.8.4 to 0.8.5.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-tabulate
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-01-11 09:19:36 +05:45
dependabot[bot] 221adf246b build(deps-dev): Bump types-toml from 0.10.1 to 0.10.3
Bumps [types-toml](https://github.com/python/typeshed) from 0.10.1 to 0.10.3.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-toml
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-01-11 09:10:44 +05:45
dependabot[bot] 8681086bfb build(deps-dev): Bump mypy from 0.930 to 0.931
Bumps [mypy](https://github.com/python/mypy) from 0.930 to 0.931.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.930...v0.931)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-01-11 09:02:39 +05:45
dependabot[bot] 6c3b1370a3 build(deps-dev): Bump types-requests from 2.26.3 to 2.27.5
Bumps [types-requests](https://github.com/python/typeshed) from 2.26.3 to 2.27.5.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-01-11 08:36:12 +05:45
Ruslan Kuprieiev 9a01d98876 odb: move gc into a separate function 2022-01-11 01:52:34 +02:00
Ruslan Kuprieiev e190433249 lint: add new ignores
mypy:

```
dvc/utils/table.py: note: In member "add_column" of class "Table":
dvc/utils/table.py:24:5: error: Argument 7 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['top'], Literal['middle'], Literal['bottom']]"  [override]
        def add_column(  # pylint: disable=arguments-differ
        ^
dvc/utils/table.py:24:5: note: This violates the Liskov substitution principle
dvc/utils/table.py:24:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
dvc/utils/table.py:24:5: error: Argument 8 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['fold'], Literal['crop'], Literal['ellipsis'], Literal['ignore']]"
 [override]
        def add_column(  # pylint: disable=arguments-differ
        ^
dvc/utils/table.py:24:5: error: Argument 12 of "add_column" is incompatible
with supertype "Table"; supertype defines the argument type as "Optional[int]"
[override]
        def add_column(  # pylint: disable=arguments-differ
        ^
Found 3 errors in 1 file (checked 129 source files)
dvc/ui/table.py:87: note: In module imported here,
dvc/compare.py:25: note: ... from here:
dvc/utils/table.py: note: In member "add_column" of class "Table":
dvc/utils/table.py:24:5: error: Argument 7 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['top'], Literal['middle'], Literal['bottom']]"  [override]
        def add_column(  # pylint: disable=arguments-differ
        ^
dvc/utils/table.py:24:5: note: This violates the Liskov substitution principle
dvc/utils/table.py:24:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
dvc/utils/table.py:24:5: error: Argument 8 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['fold'], Literal['crop'], Literal['ellipsis'], Literal['ignore']]"
 [override]
        def add_column(  # pylint: disable=arguments-differ
        ^
dvc/utils/table.py:24:5: error: Argument 12 of "add_column" is incompatible
with supertype "Table"; supertype defines the argument type as "Optional[int]"
[override]
        def add_column(  # pylint: disable=arguments-differ
        ^
Found 3 errors in 1 file (checked 128 source files)
```

pylint:

```
************* Module dvc.utils.table
dvc/utils/table.py:24:4: W0237: Parameter 'vertical' has been renamed to 'overflow' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'overflow' has been renamed to 'width' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'width' has been renamed to 'min_width' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'min_width' has been renamed to 'max_width' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'max_width' has been renamed to 'ratio' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'ratio' has been renamed to 'no_wrap' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'no_wrap' has been renamed to 'collapse' in overridden 'Table.add_column' method (arguments-renamed)
```
2022-01-11 00:47:38 +02:00
Ruslan Kuprieiev f64018b75e dvc: separate data and objects
Initial simple separation to get the process of extracting dvc-objects
and dvc-data packages out of dvc.

dvc-objects is for:
 * defining base Object (aka HashFile) and ObjectDB
 * providing tools to work with those objects (check/transfer/etc)
 * etc

dvc-data is for:
 * defining higher level objects (e.g. Tree)
 * bulding objects from the workspace (aka staging)
 * projecting objects back into the workspace (aka checkout)
 * comparing Tree objects (aka diff)
 * defining misc object databases (e.g LocalObjectDB)
 * higher-level transfer (e.g. with Tree indexes)
 * etc

For example, the objects we use in run-cache are based on dvc-objects
and not on dvc-data, as they have absolutely nothing to do with data
management.

There are still some mixups in the code that will be addressed in the
followups.
2022-01-11 00:47:38 +02:00
Gavin Li 92f714e5ca fs: test_links: create destination directory first
`dvc.fs.utils.test_links()` fails with a CacheLinkError if the target `to_path` does not already exist, as is the case when "abc.dvc" exists (referring to a directory object) and we try to run "dvc pull abc/def". To avoid this issue, fully create `to_path` before attempting to create a link in it.
2022-01-08 13:56:51 +02:00
Jon Burdo 634246f81c add tests for Path.parts 2022-01-07 13:26:45 +02:00
Jon Burdo 05b5e9dc29 handle paths ending with sep in dvc.fs.path.Path.parts 2022-01-07 13:26:45 +02:00
Paweł Redzyński a3144482ba render: vega: data processing outside of renderers
Fixes: #6943
2022-01-07 10:39:12 +01:00
Amrit GhimireandSaugat Pachhai 44a1146915 test: fix test failure in case of different default branch (#6857)
* test: fix test failure in case of different default branch

When the git config has set global `init.defaultBranch` different than master,
a few test cases were failing.
Currently, the test case is assuming the default branch to be set as
master.

This commit will introduce **initial_branch** option to git init call
which will ensure *master* branch is used as default for out test cases.

Set default git branch to main for test CI

Since main is going to be new default branch, it will let the test case
handle the scenario where default branch is set other than master.

Introduce isolation for git configuration

Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2022-01-07 11:13:04 +05:45
Saugat Pachhai (सौगात) ec3600eb8e output: don't raise RemoteCacheRequiredError on instantiation, make it lazy 2022-01-06 22:29:49 +02:00
Paweł Redzyński fe4fefd880 ls: handle broken symlink
Fixes: #7120
2022-01-06 22:12:17 +02:00
karajan1001andPeter Rowlands b6a2402d27 exp remove: refactor exp utils (#7196)
fix: #7196

1. Refactor exp_ref_by_name in `dvc/repo/experiments/utils.py`.

In previous the exp_ref_by_name walk throught all of the references to
match only one exp name. This makes multi exp name seeking to be a job
of `O(mn)` complexity, Walking though the exp ref and return all of the
exp ref with greatly reduce this complexity to `O(n)`

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2022-01-06 12:20:29 +08:00
Saugat Pachhai d90fe54df4 tests: remove tests.unit.scm
I forgot to remove it before during #6836.
2022-01-05 20:36:29 +05:45
Peter Rowlands 9e590edb70 odb: pin index pickle protocol version 2022-01-04 20:22:08 +02:00
karajan1001 6bb4889022 ui: remove None in CellT type
From https://github.com/iterative/dvc/pull/6893/files#r777784748
Table rendering does not support `None` for now, and we will change it
back after (#7167).
2022-01-04 15:41:18 +08:00
David de la Iglesia Castro 0a4958bafd TabularData: Decouple pcp from render. 2022-01-03 16:30:54 +01:00
David de la Iglesia Castro 5a9f861921 exp show: Rename html to pcp
Closes #7165
2022-01-03 16:30:54 +01:00
David de la Iglesia Castro fc1ab7d3a1 Update show.py 2022-01-03 16:14:37 +01:00
David de la Iglesia Castro 760c8fceb3 plots: Add auto_open hint 2022-01-03 16:07:23 +01:00
David de la Iglesia Castro d8e2c14927 config: Add plots.auto_open.
Option for enabling automatically open webbrowser with generated html.
Defaults False.

Closes #6091
2022-01-03 16:07:23 +01:00
dependabot[bot] a73accd071 build(deps-dev): Bump filelock from 3.4.0 to 3.4.2
Bumps [filelock](https://github.com/tox-dev/py-filelock) from 3.4.0 to 3.4.2.
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/py-filelock/blob/main/docs/changelog.rst)
- [Commits](https://github.com/tox-dev/py-filelock/compare/3.4.0...3.4.2)

---
updated-dependencies:
- dependency-name: filelock
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-01-03 20:04:58 +05:45
dependabot[bot] 69ac43f131 build(deps-dev): Bump pylint-plugin-utils from 0.6 to 0.7
Bumps [pylint-plugin-utils](https://github.com/PyCQA/pylint-plugin-utils) from 0.6 to 0.7.
- [Release notes](https://github.com/PyCQA/pylint-plugin-utils/releases)
- [Commits](https://github.com/PyCQA/pylint-plugin-utils/compare/0.6...0.7)

---
updated-dependencies:
- dependency-name: pylint-plugin-utils
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-01-03 17:36:22 +05:45
Alexander Schepanovski 9ee428ef17 Fix memory leak in Index.build_graph()
`@memoize()` memory lives as long as decorated function lives, which is
permanent in this case. Since `self` is referenced as a memory key no
`Index` will ever be freed.

Changed to use `@cached_property` whose cache lives as long as the
parent instance.
2022-01-03 13:41:10 +02:00
Peter Rowlands 0e0e1725e6 exp: handle non-exp refs in get_exact_name() 2022-01-03 15:02:19 +05:45
Saugat Pachhai (सौगात) a1609cfb10 exp: raise InvalidExpRefError properly
Before it was just left instantiated, but was not raised.
2022-01-03 15:02:19 +05:45
David de la Iglesia Castro 175ffaeb22 exp show: Remove alternating background 2022-01-03 16:38:39 +09:00
dependabot[bot] 025e15d500 build(deps-dev): Bump types-tabulate from 0.8.3 to 0.8.4
Bumps [types-tabulate](https://github.com/python/typeshed) from 0.8.3 to 0.8.4.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-tabulate
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-01-03 10:38:08 +05:45
dependabot[bot] 52a234c5b8 build(deps-dev): Bump pygments from 2.10.0 to 2.11.1
Bumps [pygments](https://github.com/pygments/pygments) from 2.10.0 to 2.11.1.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.10.0...2.11.1)

---
updated-dependencies:
- dependency-name: pygments
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-01-03 10:09:53 +05:45
Ruslan Kuprieiev 8ec1af1cf4 config: remove useless os.path.join 2021-12-31 19:03:05 +02:00
Ruslan Kuprieiev 35d364a527 repo: use fs instead of scm
SCM is used for both reading and writing operations, but in the former case -
as filesystem and in the latter - as actual scm/git. So we should distinguish
those and use fs directly, which follows the overall philosophy of being able
to use Repo on any fs.

Keeping `rev=` option for `Repo` because it is a very convenient way to get
repo instance from local path but for particular branch/tag/etc.

Pre-requisite for git/dvc/repofs refactorings.
2021-12-31 02:44:21 +02:00
David de la Iglesia Castro 9a5614d326 TabularData: Add ignore_empty flag to drop_duplicates.
Configures whether to consider missing values as relevant or not.
2021-12-29 21:33:36 +01:00
David de la Iglesia Castro 12a737d905 TabularData: Fix drop_duplicates for rich_text NaN.
When filling missing values with `ui.rich_text` (i.e. in experiments show CMD), those values were not being correctly matched against `self._fill_value`)
2021-12-29 21:33:36 +01:00
Paweł Redzyński 51a6116b92 plots: dir: handle no data in cache/remote
Fixes: iterative/cse#4
2021-12-29 13:21:37 +01:00
dependabot[bot] 3566b1941c build(deps-dev): Bump types-requests from 2.26.2 to 2.26.3
Bumps [types-requests](https://github.com/python/typeshed) from 2.26.2 to 2.26.3.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-12-29 12:53:59 +09:00
pre-commit-ci[bot] ceeda6d307 [pre-commit.ci] pre-commit autoupdate
updates:
- [github.com/pre-commit/pre-commit-hooks: v4.0.1 → v4.1.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.0.1...v4.1.0)
2021-12-28 15:01:38 +08:00
karajan1001 98d7f785e9 exp show: remove a useless function
The function `_raise_error_if_all_disabled` is copyed wrongly to this
place during sub command refactoring. It used to be used in `exp gc`,
can be deleted from this function.
2021-12-27 20:35:07 +01:00
Ruslan Kuprieiev 04e93da0bd fs: remove unused RemoteCmdError 2021-12-23 19:02:44 +02:00
Ruslan Kuprieiev 72aaca4343 odb: move verify option to config
These options are part of the default config for each particular remote.
Plugins will define their default config on theirown, but for now we keep
those in one config schema.
2021-12-23 12:49:16 +02:00
dependabot[bot] 60cb8322d9 build(deps-dev): Bump mypy from 0.921 to 0.930
Bumps [mypy](https://github.com/python/mypy) from 0.921 to 0.930.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.921...v0.930)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-12-23 09:50:23 +05:45
dependabot[bot] b0b9a74207 build(deps-dev): Bump wheel from 0.37.0 to 0.37.1
Bumps [wheel](https://github.com/pypa/wheel) from 0.37.0 to 0.37.1.
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](https://github.com/pypa/wheel/compare/0.37.0...0.37.1)

---
updated-dependencies:
- dependency-name: wheel
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-12-23 09:50:05 +05:45
Saugat Pachhai (सौगात) 32fcc73078 deps: explicitly specify pyarrow for hdfs 2.9.3 2021-12-22 16:45:12 +02:00
Saugat Pachhai (सौगात) e3596178d4 deps: move requests-kerberos to webhdfs_kerberos extras instead of webhdfs 2021-12-22 16:44:52 +02:00
Saugat Pachhai (सौगात) ba40d91de6 fs.hdfs: skip port if not available 2021-12-22 16:43:56 +02:00
dependabot[bot] 02cfbcfc32 build(deps-dev): Bump mypy from 0.920 to 0.921
Bumps [mypy](https://github.com/python/mypy) from 0.920 to 0.921.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.920...v0.921)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-12-22 12:45:02 +02:00
dependabot[bot] 96acb64fec build(deps): Bump scmrepo from 0.0.6 to 0.0.7
Bumps [scmrepo](https://github.com/iterative/scmrepo) from 0.0.6 to 0.0.7.
- [Release notes](https://github.com/iterative/scmrepo/releases)
- [Commits](https://github.com/iterative/scmrepo/compare/0.0.6...0.0.7)

---
updated-dependencies:
- dependency-name: scmrepo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

tests: fix invalid set_ref in exp func test
2021-12-22 17:52:30 +09:00
David de la Iglesia Castro 1507a6ed99 Update benchmarks.yaml 2021-12-21 09:03:19 +05:45
Saugat Pachhai 6034f0e36a s3: remove explicit dependency on aiobotocore; simplify constraints
Hoping to fix `dvc[s3]` installations in pipenv. Pipenv's resolver is in loop, backtracking ruamel.yaml from 0.17.17 to 0.17.11 continuously. I think this has to do
more with `s3`'s dependency than `ruamel.yaml` as it only fails here. And, we are specifying dependency in 1.0.1 whereas aiobotocore has released 2.0 versions which is what is being installed, so it's redundant anyway. I am hopeful that this may help resolver with the simplification in constraints.
2021-12-20 14:32:42 +05:45