Commit Graph
112 Commits
Author SHA1 Message Date
skshetry 6d80b72434 lint: run lint checks on all platforms (#8832) 2023-01-17 10:51:57 +05:45
skshetry d87cdba6d8 gha: remove env vars SHELL 2023-01-10 10:19:29 +05:45
Saugat Pachhai (सौगात) 81f37be9d6 ci: timeout tests if it takes >5min
It most likely means it's frozen and unlikely to resume again. We'd want
to see where it froze, so this will provide us with a traceback.
2023-01-06 20:16:20 +05:45
skshetry ea8a4a3db4 GHA: remove unused env from tests workflow 2022-12-27 10:26:29 +05:45
skshetry a6bfc7db4e ci: test with 3.11 on all OSes (#8543)
* ci: test with 3.11 on all OSes

* Update tests.yaml

* Update tests.yaml
2022-11-09 12:05:27 +00:00
skshetry d2265f36ba ci: run only single macOS job (#8509)
* ci: run only single macOS job

* ci: use alls-green@release gate check
2022-11-01 14:30:46 +05:45
skshetry 4b720e5820 remove .codecov.yml and MANIFEST.in, move .coveragerc to pyproject.toml (#8500)
* remove .codecov.yml and MANIFEST.in, move .coveragerc to pyproject.toml

* remove scripts/pypi_upload.sh
2022-11-01 10:11:55 +05:45
Step Security Bot c8e39523c3 [StepSecurity] ci: Harden GitHub Actions (#8496)
Signed-off-by: StepSecurity Bot <[email protected]>
2022-10-31 23:27:18 +05:45
skshetry 7648e715f8 move project config to pyproject.toml (#8493)
* move config to pyproject.toml

* ci: use pyproject.toml for cache key

* pyproject.toml: use recursive dependencies for dev and all

* Update pyproject.toml
2022-10-31 22:43:48 +05:45
skshetry ca5ad6457f ci: cache lint deps, decrease test timeout (#8494) 2022-10-31 22:15:54 +05:45
skshetry 6df70d388e update pre-commit; add some more hooks (#8492)
* update pre-commit; add some more hooks

* Apply suggestions from code review
2022-10-31 21:04:11 +05:45
skshetry f6b1f01a5c aiohttp: remove AIOHTTP_NO_EXTENSIONS envvar (#8353)
* aiohttp: remove AIOHTTP_NO_EXTENSIONS envvar

aiohttp now provides wheel for 3.11

* advertise 3.11 support
2022-09-23 16:23:37 +05:45
skshetry b5f20f8467 use native concurrency (#8320) 2022-09-19 10:23:55 +05:45
skshetry f854fc6365 try to split testsuites into two (#8297)
try to split tests
2022-09-15 22:11:02 +05:45
skshetry e5eec59cb8 test on 3.11 (#8196)
We don't have aiohttp support yet, due to which we have to install
without cythonized extension.

Similarly, there has been some changes in how dataclasses disallows some
mutable arguments to be set as a class property, due to which
`hydra.conf` fails to import in 3.11.

See https://github.com/python/cpython/pull/29867. For now, I have
patched the module and avoid importing as a pytest-plugin.

dvc-hdfs/pyarrow is not there yet, we'll usually see a release 2-3
months after official reelase.

dvc-hdfs is skipped in 3.11 in extras_requires. `--set-params` will
return a nice error message if hydra cannot be imported in 3.11. I
decided to only patch that in tests.
2022-08-29 19:58:17 +05:45
dependabot[bot] 685a2d5579 build(deps): Bump styfle/cancel-workflow-action from 0.9.1 to 0.10.0
Bumps [styfle/cancel-workflow-action](https://github.com/styfle/cancel-workflow-action) from 0.9.1 to 0.10.0.
- [Release notes](https://github.com/styfle/cancel-workflow-action/releases)
- [Commits](https://github.com/styfle/cancel-workflow-action/compare/0.9.1...0.10.0)

---
updated-dependencies:
- dependency-name: styfle/cancel-workflow-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-06-28 09:15:05 +02:00
dependabot[bot] b094839210 build(deps): Bump actions/setup-python from 3 to 4
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 3 to 4.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-06-13 20:28:51 +05:45
dependabot[bot] 88d3582d40 build(deps): Bump pre-commit/action from 2.0.3 to 3.0.0
Bumps [pre-commit/action](https://github.com/pre-commit/action) from 2.0.3 to 3.0.0.
- [Release notes](https://github.com/pre-commit/action/releases)
- [Commits](https://github.com/pre-commit/action/compare/v2.0.3...v3.0.0)

---
updated-dependencies:
- dependency-name: pre-commit/action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-06-06 04:20:08 -07:00
Ruslan Kuprieiev 844f77b4f9 dvc: drop python 3.7
Also upgraded with `pyupgrade --py38-plus`

Fixes #7708
2022-05-30 16:51:35 +03:00
dependabot[bot] 1e0a4a26eb build(deps): Bump codecov/codecov-action from 2.1.0 to 3
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2.1.0 to 3.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2.1.0...v3)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-04-06 17:54:18 +02:00
Saugat Pachhai (सौगात) 565a813e2a setup.cfg: tests: try to remove more dependencies 2022-03-31 13:36:43 +05:45
dependabot[bot] 290a593322 build(deps): Bump actions/cache from 2.1.7 to 3
Bumps [actions/cache](https://github.com/actions/cache) from 2.1.7 to 3.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2.1.7...v3)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-03-22 10:32:12 +08:00
dependabot[bot] ab024a47b3 build(deps): Bump actions/checkout from 2.4.0 to 3
Bumps [actions/checkout](https://github.com/actions/checkout) from 2.4.0 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2.4.0...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-03-03 15:05:39 +08:00
dependabot[bot] a2e25cf449 build(deps): Bump actions/setup-python from 2 to 3
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 3.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-03-01 17:30:52 +09:00
Amrit GhimireandSaugat Pachhai 44a1146915 test: fix test failure in case of different default branch (#6857)
* test: fix test failure in case of different default branch

When the git config has set global `init.defaultBranch` different than master,
a few test cases were failing.
Currently, the test case is assuming the default branch to be set as
master.

This commit will introduce **initial_branch** option to git init call
which will ensure *master* branch is used as default for out test cases.

Set default git branch to main for test CI

Since main is going to be new default branch, it will let the test case
handle the scenario where default branch is set other than master.

Introduce isolation for git configuration

Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2022-01-07 11:13:04 +05:45
Ruslan Kuprieiev 0bf56fe29b rename master -> main 2021-12-15 22:19:36 +02:00
Ruslan Kuprieiev 42641ddd20 gha: remove remote leftovers
Followup for #7087
2021-12-03 23:11:22 +02:00
Paweł Redzyński 57c6b1036f gha: allow builds for new 3.9 versions
Closes: #6975
2021-11-25 14:40:40 +02:00
dependabot[bot] 13ca959e98 build(deps): Bump actions/cache from 2.1.6 to 2.1.7
Bumps [actions/cache](https://github.com/actions/cache) from 2.1.6 to 2.1.7.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2.1.6...v2.1.7)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-24 08:47:13 +05:45
Saugat Pachhai (सौगात) 46289182d2 tests: run on Windows with Python 3.10
pygit2 wheels should be now available.
Had to skip pytest-docker for Windows + Python 3.10 due to
a dependency issue with it's dependency, see:
https://github.com/docker/docker-py/issues/2902.
2021-11-20 18:52:31 +02:00
Paweł Redzyński ab56e4bb9b gha: limit python version to 3.9.7
Related: https://bugs.python.org/issue45235
Closes: #6956
2021-11-11 01:14:03 +02:00
dependabot[bot] b927337a30 build(deps): bump actions/checkout from 2.3.5 to 2.4.0
Bumps [actions/checkout](https://github.com/actions/checkout) from 2.3.5 to 2.4.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2.3.5...v2.4.0)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-03 12:47:22 +02:00
Saugat Pachhai 800fbc7ad0 Use -n=auto in tests
Linux/Windows machines have only 2 cores, whereas macOS machines have 3 cores. Maybe we are stretching available resources by 2x, let's see if `-n=auto` improves test freezes.
2021-10-22 08:46:55 +05:45
Peter Rowlands (변기호) 73f2fa8c93 git: use asyncssh as dulwich SSH vendor (#6797)
* git: support client kwargs for auth in remote methods

* tests: use docker for git/ssh server

* move dulwich backend into subdir

* add base sync<->async support

* dulwich: add initial asyncssh SSH vendor

* dulwich: use asyncssh as the default SSH vendor

* add test for git over ssh

* test read (fetch) over ssh

* ci: re-enable ssh tests

temporary workaround for #6788, #6789

* move async stubs into dvc.scm.asyn

* tests: use sshfs and not dvc.fs.ssh

* allow explicit 0-length read in asyncssh vendor
2021-10-21 14:36:45 +09:00
Saugat Pachhai 0c41a25bbe move some configs to pyproject.toml (#6834) 2021-10-19 19:47:51 +03:00
dependabot[bot] 4ed4766692 build(deps): Bump actions/checkout from 2.3.4 to 2.3.5 (#6817)
Bumps [actions/checkout](https://github.com/actions/checkout) from 2.3.4 to 2.3.5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2.3.4...v2.3.5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-18 19:43:41 +05:45
Ruslan Kuprieiev e9a7e0657d dvc: drop 3.6 support (#6792)
* dvc: drop 3.6 support

Considering that 3.6 has ~2 months until EOL, and our deps are
dropping support for it already (e.g. pygit2), we are droppipping
it as well. This will reduce the load on our test CI as well, since
we've started supporting 3.10 recently.

* setup.cfg: adjust requirements

* daemon: remove 3.6 logic

* tests: utils: remove pre 3.7 logic
2021-10-12 18:53:28 +03:00
Saugat PachhaiandRuslan Kuprieiev 8abda93e3e use setuptools_scm and build_requires (#6775)
* use setuptools_scm and build_requires

* remove kwargs from setup.py

* fix tests; fetch in depth in github actions

* use build to build sdist/wheel

* try updating pip and setuptools

* update pip

* remove or unpin wheel

* bring back the version in pyinstaller

* Update scripts/build.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-10-12 17:58:14 +03:00
Ruslan Kuprieiev 53255c22b3 gha: tests: temporarily disable some remote tests (#6789)
* gha: tests: temporarily disable some remote tests

See #6788

* machine: upgrade tpi version

```
dvc/machine/backend/terraform.py:28:19: E1101: Instance of 'TerraformBackend' has no '_default_resource' member; maybe 'default_resource'? (no-member)
```

* tests: remotes: require --s3 for s3 tests
2021-10-12 17:26:47 +03:00
Paweł Redzyński 8580e66d60 gha: move cancelaltion into other workflows (#6771) 2021-10-08 18:03:07 +03:00
Saugat Pachhai f628f670c9 advertise Python 3.10 support (#6772)
* advertise Python 3.10 support

Pygit2 does not have windows wheels yet and no pyarrow
for 3.10. But these platform-dependent things and
remote-dependent things should not stop us from
advertising support as we are pure-python based
package and we cannot claim support for all of our
dependencies.

* set pygit2 for Python 3.6 to <1.7
2021-10-08 15:50:05 +03:00
Saugat Pachhai c5415342d0 move requirements to setup.cfg (#6758)
* move requirements to setup.cfg

* hash based on setup.cfg

* do not break webhdfs installation
2021-10-08 07:00:14 +05:45
Paweł Redzyński dd86076ba6 gha: cancel previous workflows (#6756) 2021-10-07 17:59:44 +03:00
Saugat Pachhai ee823a59b9 Run on Python3.10 (#6745) 2021-10-05 16:46:53 +05:45
Saugat Pachhai 46ad8745d3 Use latest pygit2 from master for 3.10 (#6723)
* Use latest pygit2 from master for 3.10

* ci: update brew
2021-10-01 11:48:11 +05:45
Ruslan Kuprieiev e81c88e501 gha: increase timeouts (#6645) 2021-09-19 23:47:15 +03:00
Saugat Pachhai e049c4bab1 Set timeout for tests back to 30
Reverts 93a200c
2021-09-15 19:10:52 +05:45
Ruslan Kuprieiev 93a200cf5f gha: increase timeout 2021-09-14 18:37:10 +03:00
dependabot[bot] 1a1294848b build(deps): Bump codecov/codecov-action from 2.0.3 to 2.1.0 (#6606)
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2.0.3 to 2.1.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.3...v2.1.0)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-09-14 08:46:38 +05:45
Saugat Pachhai f9bb38b96f Set timeout for jobs in Github Actions (#6545)
* Set timeout for jobs in Github Actions

Sometimes tests are getting frozen(most likely an issue with `pytest-xdist`), and sometimes the pip resolver is slow due to some weird constraints set by us or one of our dependencies.

The default timeout is 360 minutes (6hrs), but our test completes on ~18 minutes at maximum, so I think 30mins should be enough to account for occasional CI slowdown.

Also keeping the total job timeout at 45 mins, we usually require just 4 mins of extra time to do installation and setup but giving it 11 more minutes just in case.

Similarly, `lint` completes in <5 mins usually, but giving it a 10 mins timeout.

This should give us early feedback on the issue.

* Update setup.cfg

* Update setup.cfg
2021-09-07 14:08:22 +05:45