Commit Graph
6946 Commits
Author SHA1 Message Date
Ruslan Kuprieiev e9a7e0657d dvc: drop 3.6 support (#6792)
* dvc: drop 3.6 support

Considering that 3.6 has ~2 months until EOL, and our deps are
dropping support for it already (e.g. pygit2), we are droppipping
it as well. This will reduce the load on our test CI as well, since
we've started supporting 3.10 recently.

* setup.cfg: adjust requirements

* daemon: remove 3.6 logic

* tests: utils: remove pre 3.7 logic
2021-10-12 18:53:28 +03:00
Saugat PachhaiandRuslan Kuprieiev 8abda93e3e use setuptools_scm and build_requires (#6775)
* use setuptools_scm and build_requires

* remove kwargs from setup.py

* fix tests; fetch in depth in github actions

* use build to build sdist/wheel

* try updating pip and setuptools

* update pip

* remove or unpin wheel

* bring back the version in pyinstaller

* Update scripts/build.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-10-12 17:58:14 +03:00
Ruslan Kuprieiev 53255c22b3 gha: tests: temporarily disable some remote tests (#6789)
* gha: tests: temporarily disable some remote tests

See #6788

* machine: upgrade tpi version

```
dvc/machine/backend/terraform.py:28:19: E1101: Instance of 'TerraformBackend' has no '_default_resource' member; maybe 'default_resource'? (no-member)
```

* tests: remotes: require --s3 for s3 tests
2021-10-12 17:26:47 +03:00
Paweł RedzyńskiandJorge Orpinel 6da15629c4 plots: modify: add note on modyfying image type plots (#6742)
* plots: modify: add note on modyfying image type plots

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

Co-authored-by: Jorge Orpinel <[email protected]>
2021-10-12 16:08:49 +03:00
Ruslan Kuprieiev 5cfe61928e tree: use oids instead of objects (#6785) 2021-10-12 05:39:10 +03:00
Ruslan Kuprieiev 32467ada32 objects: extract size/nfiles to Meta (#6784)
* stage: capture metadata

* tree: capture metadata

* objects: extract size/nfiles from HashInfo/HashFile/Tree to Meta
2021-10-12 03:06:18 +03:00
pre-commit-ci[bot] 5cbaa8d98e [pre-commit.ci] pre-commit autoupdate (#6782)
updates:
- [github.com/pre-commit/mirrors-mypy: v0.910 → v0.910-1](https://github.com/pre-commit/mirrors-mypy/compare/v0.910...v0.910-1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-10-12 02:27:56 +03:00
Ruslan Kuprieiev 4762c38ff7 dvc: bump to 2.8.1 2.8.1 2021-10-11 16:09:51 +03:00
Ruslan Kuprieiev 66acbc892a setup.cfg: use text/x-rst instead of text/rst
Per https://packaging.python.org/specifications/core-metadata/#description-content-type
2021-10-11 16:09:29 +03:00
Ruslan Kuprieiev d2edbae7e3 dvc: bump to 2.8.0 2.8.0 2021-10-11 15:47:58 +03:00
Batuhan Taskaya be0c827b95 fs: add basic type annotations to FsspecWrapper (#6779) 2021-10-11 15:15:56 +03:00
Paweł Redzyński d6ecef7169 plots: fix wrong link (#6693) 2021-10-11 12:22:20 +02:00
eggqq007andZhenyu Gao e8b649a75f Fix exp show --sort-by issue #6675 (#6764)
Co-authored-by: Zhenyu Gao <[email protected]>
2021-10-11 16:46:38 +09:00
Batuhan Taskaya 8767f42f39 gs: use 3-char prefixes (#6759)
Resolves #6691. Normally it should have inherited this from the `ObjectFSWrapper`, but this seems like a leftover.
2021-10-11 16:16:17 +09:00
Paweł Redzyński 8580e66d60 gha: move cancelaltion into other workflows (#6771) 2021-10-08 18:03:07 +03:00
Saugat Pachhai c24ae2b0c3 exp init: show defaults in help message (#6763) 2021-10-08 20:14:12 +05:45
Saugat Pachhai 4e17279440 exp init: add config support (#6761) 2021-10-08 20:14:01 +05:45
Batuhan Taskaya 50cdf4f213 fsspec: normalize checksum() to return string (#6767) 2021-10-08 16:57:37 +03:00
Saugat Pachhai f628f670c9 advertise Python 3.10 support (#6772)
* advertise Python 3.10 support

Pygit2 does not have windows wheels yet and no pyarrow
for 3.10. But these platform-dependent things and
remote-dependent things should not stop us from
advertising support as we are pure-python based
package and we cannot claim support for all of our
dependencies.

* set pygit2 for Python 3.6 to <1.7
2021-10-08 15:50:05 +03:00
Batuhan Taskaya d7489bb65f hdfs: enable mocked hdfs tests by default (#6766) 2021-10-08 12:42:41 +02:00
dependabot[bot]andPaweł Redzyński 5a4f4d3a8e build(deps): Bump filelock from 3.2.0 to 3.3.0 (#6737)
* build(deps): Bump filelock from 3.2.0 to 3.3.0

Bumps [filelock](https://github.com/tox-dev/py-filelock) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Commits](https://github.com/tox-dev/py-filelock/compare/3.2.0...3.3.0)

---
updated-dependencies:
- dependency-name: filelock
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* tests: remotes: ignore abstract class instantiation for filelock

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Paweł Redzyński <[email protected]>
2021-10-08 15:36:33 +05:45
dependabot[bot] 2c24b24562 build(deps-dev): Bump types-requests from 2.25.6 to 2.25.9 (#6770)
Bumps [types-requests](https://github.com/python/typeshed) from 2.25.6 to 2.25.9.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-08 14:59:09 +05:45
dependabot[bot] df9f76c2c0 build(deps-dev): Bump types-paramiko from 0.1.9 to 2.7.1 (#6769)
Bumps [types-paramiko](https://github.com/python/typeshed) from 0.1.9 to 2.7.1.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-paramiko
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-08 14:58:19 +05:45
dependabot[bot] 0db3ece060 build(deps-dev): Bump types-toml from 0.1.5 to 0.10.0 (#6768)
Bumps [types-toml](https://github.com/python/typeshed) from 0.1.5 to 0.10.0.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-toml
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-08 14:58:08 +05:45
Saugat Pachhai 7c437cc349 make dependabot updates work (#6760)
Dependabot was throwing Illformed requirements `"< '3.10'"` for the
hdfs. But I noticed that it did accept other requirements where we
do similar. So it seems the indentation fixes it.

Even then, dependabot did not update any dependencies. I removed
setup.py file, after which immediately it started creating PRs.
So, I recknoed that it must have been due to some mechanism
in place from dependabot to disallow execution, so I tried to
remove few blocks of code from setup.py. And, I was eventually
able to figure out that it was from the block where we import version.

So, I added a try-except block and it magically again worked. :)

Let's see how it goes.
2021-10-08 14:46:47 +05:45
Saugat Pachhai 3ed5efceb7 move defaults to the class level (#6762) 2021-10-08 12:18:28 +05:45
dependabot[bot]andPaweł Redzyński d33df60e41 build(deps): Bump dvclive from 0.3.0 to 0.4.0 (#6753)
* build(deps): Bump dvclive from 0.3.0 to 0.4.0

Bumps [dvclive](https://github.com/iterative/dvclive) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/iterative/dvclive/releases)
- [Commits](https://github.com/iterative/dvclive/compare/0.3.0...0.4.0)

---
updated-dependencies:
- dependency-name: dvclive
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* live: adjust tests to dvclive==0.4.0

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Paweł Redzyński <[email protected]>
2021-10-08 07:02:54 +05:45
Saugat Pachhai c5415342d0 move requirements to setup.cfg (#6758)
* move requirements to setup.cfg

* hash based on setup.cfg

* do not break webhdfs installation
2021-10-08 07:00:14 +05:45
Paweł Redzyński dd86076ba6 gha: cancel previous workflows (#6756) 2021-10-07 17:59:44 +03:00
Ruslan Kuprieiev 7dc3859400 pyinstaller: hooks: remove webhdfs 2021-10-07 17:30:22 +03:00
Batuhan Taskaya 4f3b28e268 webhdfs: migrate to fsspec (#6662)
* webhdfs: migrate to fsspec

* skip the tests that can't use hdfs
2021-10-07 17:02:31 +03:00
Ruslan Kuprieiev 32f435c47f gha: benchmarks: fix pytest flags (#6749)
Kudos @skshetry for catching.
2021-10-06 17:07:41 +03:00
Batuhan Taskaya 054feae778 hdfs: migrate to fsspec (#6604)
* hdfs: migrate to fsspec

* normalize windows paths when delegating back to fsspec
2021-10-06 15:44:27 +03:00
Saugat Pachhai 2abe515391 Pin dvclive to 0.3.0 2021-10-06 14:28:36 +05:45
eggqq007andZhenyu Gao 1eb89a2dc0 command: Should not consume 'print' to output default remote info (#6650)
1. should not consume 'print' to output default remote info,
   otherwise the --quiet would not work.

Co-authored-by: Zhenyu Gao <[email protected]>
2021-10-06 09:27:47 +05:45
eggqq007andZhenyu Gao 9c300a0a1c command: output prettify json in tty (#6743)
1. update diff.py,experiments.py,metrics.py,params.py,status.py to
print prettify json
2. rewrite write_json in __init__.py to accept default function
3. add new unit test for write_json in tests/unit/ui/test_console.py
4. Fix #6712

Co-authored-by: Zhenyu Gao <[email protected]>
2021-10-05 20:19:56 +05:45
Ruslan Kuprieiev 7b68a6f0ca gha: run benchmarks on PRs (#6733) 2021-10-05 14:18:43 +03:00
Saugat Pachhai ee823a59b9 Run on Python3.10 (#6745) 2021-10-05 16:46:53 +05:45
David de la Iglesia CastroandSaugat Pachhai 16f404b16a Fix dvclive running subdir (#6740)
* Fix dvclive running subdir

* Apply suggestions from code review

Co-authored-by: Saugat Pachhai <[email protected]>

* Applied suggestions

* Update tests/func/test_live.py

Co-authored-by: Saugat Pachhai <[email protected]>
2021-10-05 13:16:33 +05:45
Saugat Pachhai 0eda4bd91c exp init: only ask for that are not provided in an interactive mode (#6739) 2021-10-04 14:33:08 +05:45
Saugat Pachhai a098a50c8d move exp init logic to dvc.repo.experiments.init (#6738) 2021-10-04 12:26:34 +05:45
dependabot[bot] f57a378c0c build(deps): Bump jaraco-windows from 5.6.0 to 5.7.0 (#6735)
Bumps [jaraco-windows](https://github.com/jaraco/jaraco.windows) from 5.6.0 to 5.7.0.
- [Release notes](https://github.com/jaraco/jaraco.windows/releases)
- [Changelog](https://github.com/jaraco/jaraco.windows/blob/main/CHANGES.rst)
- [Commits](https://github.com/jaraco/jaraco.windows/compare/v5.6.0...v5.7.0)

---
updated-dependencies:
- dependency-name: jaraco-windows
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-04 10:12:47 +05:45
dependabot[bot] 3e0a50e52e build(deps): Bump pytest-cov from 2.12.1 to 3.0.0 (#6736)
Bumps [pytest-cov](https://github.com/pytest-dev/pytest-cov) from 2.12.1 to 3.0.0.
- [Release notes](https://github.com/pytest-dev/pytest-cov/releases)
- [Changelog](https://github.com/pytest-dev/pytest-cov/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest-cov/compare/v2.12.1...v3.0.0)

---
updated-dependencies:
- dependency-name: pytest-cov
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-04 09:31:27 +05:45
Guillaume COTER ec72866eb9 Bump typing_extensions to >=3.7.4 (#6731) (#6732) 2021-10-03 20:10:28 +05:45
Batuhan Taskaya e53d390ff9 http: explicitly ignore all http-related errors (from server side to client connection drops) (#6728) 2021-10-02 20:42:23 +03:00
Batuhan Taskaya dd330ac592 setup: bump fsspec/s3fs/gcsfs (#6727) 2021-10-02 18:54:09 +02:00
pwoolvett 4ccb712b6d build(setup): avoid leaking tests package into site-packages (#6585)
Signed-off-by: Pablo Woolvett <[email protected]>
2021-10-02 16:46:45 +03:00
Saugat Pachhai ecee2d3522 Move static metadata to setup.cfg (#6581) 2021-10-02 16:08:30 +03:00
20bb6ddf3c doctor: fixes unknown cache directory error on some windows mapped network shares (#6364)
* doctor: fixes unknown cache director error on some windows mapped network shares

On Windows, resolve calls os.path._getfinalpathname, which calls GetFinalPathNameByHandle.
This starts from the final resolved path of the file in the NT namespace (e.g. \Device\Mup\server\share...)
and works backward as required to get the VOLUME_NAME_DOS. If the final path is a UNC share (i.e. \Device\Mup),
We fix this by calling resolve() also on the partition mountpoints
We also take the opportunity to make checks in case a PermissionError is returned to us

Fixes github.com/iterative/dvc/issues/6305

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* doctor: fixes unknown cache director error on some windows mapped network shares

On Windows, resolve calls os.path._getfinalpathname, which calls GetFinalPathNameByHandle.
This starts from the final resolved path of the file in the NT namespace (e.g. \Device\Mup\server\share...)
and works backward as required to get the VOLUME_NAME_DOS. If the final path is a UNC share (i.e. \Device\Mup),
We fix this by calling resolve() also on the partition mountpoints
We also take the opportunity to make checks in case a PermissionError is returned to us

Fixes #6305

Co-authored-by: CHETOUAN <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: CHETOUAN <[email protected]>
2021-10-02 16:06:42 +03:00
Saugat Pachhai e58fcd864b [WIP] exp init: introduce --with={default,live} flag (#6716)
* exp init: introduce `--with={default,live}` flag

This flag will select the different mode of outputs of stage.
With `--with=live`, it'll enable `live` setting in the stage.
On interactive mode, it'll always ask user if they want to
use live (unless `--with=live` is specified), and then it'll
skip metrics/plots question and moves on to ask for path to
dvclive path.

Also, if live setting is enabled in the stage, it'll set the
`models` to be a checkpoint output.

* s/with/type and don't ask for live interactively
2021-10-02 16:12:44 +05:45