Saugat Pachhai
f9bb38b96f
Set timeout for jobs in Github Actions ( #6545 )
...
* Set timeout for jobs in Github Actions
Sometimes tests are getting frozen(most likely an issue with `pytest-xdist`), and sometimes the pip resolver is slow due to some weird constraints set by us or one of our dependencies.
The default timeout is 360 minutes (6hrs), but our test completes on ~18 minutes at maximum, so I think 30mins should be enough to account for occasional CI slowdown.
Also keeping the total job timeout at 45 mins, we usually require just 4 mins of extra time to do installation and setup but giving it 11 more minutes just in case.
Similarly, `lint` completes in <5 mins usually, but giving it a 10 mins timeout.
This should give us early feedback on the issue.
* Update setup.cfg
* Update setup.cfg
2021-09-07 14:08:22 +05:45
Ruslan Kuprieiev
6287b13ba3
gha: allow running tests on other branches ( #6520 )
...
Simplifies development for all contributors.
2021-09-02 17:55:54 +03:00
Peter Rowlands (변기호)
4ee98299ee
dvc machine: implement initial terraform support (#6462 )
...
* tpi: add naive terraform-provider-iterative wrapper
* executor: add naive backend/terraform wrapper
* setup: add dvc[terraform] install extras
* config: add feature.executor feature flag
* fix named executor config sections
* executor: add poc add/modify/remove/init/destroy commands
* relax python-terraform requirement
* ci: install terraform extras
* tpi: support .tf.json format
* s/executor/machine
* terraform: use separate dirs per machine
* s/init/create/
* reserve local machine name
* add initial ssh skeleton
* add initial machine unit tests
* add initial machine command tests
* add tests for add/remove
* enable feature by default in test env
* close temp pemfile object so it can be used on windows
* cleanup pemfile syntax
2021-08-27 12:46:24 +03:00
dependabot[bot]
ceb0964fcf
build(deps): Bump codecov/codecov-action from 2.0.2 to 2.0.3 ( #6487 )
...
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action ) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/codecov/codecov-action/releases )
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md )
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.2...v2.0.3 )
---
updated-dependencies:
- dependency-name: codecov/codecov-action
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <[email protected]
2021-08-25 08:22:52 +05:45
Saugat Pachhai
1bb7f39e14
add support for Python 3.10 ( #6422 )
...
* add support for Python 3.10
pyarrow is not available, so we are skipping tests for it.
And, restricting `hdfs` extras to be not installed in Python3.10.
Pathlib removed `_init` method and `init` parameter, so just working
around that. It's not a big change.
* skip windows on 3.10; update pip/wheel/setuptools during install
pywin32 is not yet available for 3.10 Python, so skipping those for now
* install libgit2 with Python3.10 as pygit2 wheel is not available
* install libgit2
* only run on macOS, libgit2-1.1 not available on ubuntu
2021-08-16 16:24:28 +05:45
dependabot[bot]
b294bca383
build(deps): bump codecov/codecov-action from 2.0.1 to 2.0.2 ( #6361 )
...
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action ) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/codecov/codecov-action/releases )
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md )
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.1...v2.0.2 )
---
updated-dependencies:
- dependency-name: codecov/codecov-action
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <[email protected]
2021-07-28 10:21:10 +03:00
dependabot[bot]
005bca1da6
build(deps): bump codecov/codecov-action from 1.5.2 to 2.0.1 ( #6334 )
...
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action ) from 1.5.2 to 2.0.1.
- [Release notes](https://github.com/codecov/codecov-action/releases )
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md )
- [Commits](https://github.com/codecov/codecov-action/compare/v1.5.2...v2.0.1 )
---
updated-dependencies:
- dependency-name: codecov/codecov-action
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <[email protected]
2021-07-20 14:06:50 +03:00
Gao and Ruslan Kuprieiev
93f36d9351
oss: migrate to ossfs ( #6307 )
...
* Migrate to ossfs
fix 6177
* Add verification to object stored on oss
* Change the oss remote's dependency.
pass all of the tests
* Performance optimaztion and ossfs version update
* Remove the argument to set the resumable get/put
1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default
* Use a new api for performance optimization
* Without filesize progress bar version
* With progress bar version
* Bump and add comments
1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger
* Change the name of kwrags following parent class.
* Bump to new ossfs and rewrite upload_fobj
Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj
* Optimization on rm_files
* Some optimization follow the old version.
* Update dvc/fs/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected] [email protected] [email protected]
2021-07-13 15:27:34 +03:00
dependabot[bot]
57c728269d
build(deps): bump codecov/codecov-action from 1.5.0 to 1.5.2 ( #6139 )
...
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action ) from 1.5.0 to 1.5.2.
- [Release notes](https://github.com/codecov/codecov-action/releases )
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md )
- [Commits](https://github.com/codecov/codecov-action/compare/v1.5.0...v1.5.2 )
---
updated-dependencies:
- dependency-name: codecov/codecov-action
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <[email protected]
2021-06-09 23:30:31 +05:45
dependabot[bot]
bf5c28b71a
build(deps): bump rtCamp/action-slack-notify from 2.1.3 to 2.2.0 ( #6128 )
...
Bumps [rtCamp/action-slack-notify](https://github.com/rtCamp/action-slack-notify ) from 2.1.3 to 2.2.0.
- [Release notes](https://github.com/rtCamp/action-slack-notify/releases )
- [Commits](https://github.com/rtCamp/action-slack-notify/compare/v2.1.3...v2.2.0 )
---
updated-dependencies:
- dependency-name: rtCamp/action-slack-notify
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <[email protected]
2021-06-07 14:04:32 +05:45
dependabot[bot]
efda015de4
build(deps): bump actions/cache from 2.1.5 to 2.1.6 ( #6073 )
...
Bumps [actions/cache](https://github.com/actions/cache ) from 2.1.5 to 2.1.6.
- [Release notes](https://github.com/actions/cache/releases )
- [Commits](https://github.com/actions/cache/compare/v2.1.5...v2.1.6 )
Signed-off-by: dependabot[bot] <[email protected]
2021-05-28 14:43:23 +05:45
dependabot[bot]
0ca08d2fea
build(deps): bump rtCamp/action-slack-notify from 2 to 2.1.3 ( #5993 )
...
Bumps [rtCamp/action-slack-notify](https://github.com/rtCamp/action-slack-notify ) from 2 to 2.1.3.
- [Release notes](https://github.com/rtCamp/action-slack-notify/releases )
- [Commits](https://github.com/rtCamp/action-slack-notify/compare/v2...v2.1.3 )
Signed-off-by: dependabot[bot] <[email protected]
2021-05-12 13:06:50 +05:45
Saugat Pachhai
eabf6ddf27
run gdrive tests in a single job, run all on schedule ( #5974 )
2021-05-06 17:09:34 +05:45
Saugat Pachhai
c2f1a1e162
Notify CI failures to Slack ( #5970 )
2021-05-05 18:37:33 +05:45
dependabot[bot]
c2734a2cd7
build(deps): bump codecov/codecov-action from v1.4.1 to v1.5.0 ( #5957 )
...
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action ) from v1.4.1 to v1.5.0.
- [Release notes](https://github.com/codecov/codecov-action/releases )
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md )
- [Commits](https://github.com/codecov/codecov-action/compare/v1.4.1...a1ed4b322b4b38cb846afb5a0ebfa17086917d27 )
Signed-off-by: dependabot[bot] <[email protected]
2021-05-04 17:45:01 +03:00
dependabot[bot]
af6921e9ac
build(deps): bump pre-commit/action from v2.0.2 to v2.0.3 ( #5856 )
...
Bumps [pre-commit/action](https://github.com/pre-commit/action ) from v2.0.2 to v2.0.3.
- [Release notes](https://github.com/pre-commit/action/releases )
- [Commits](https://github.com/pre-commit/action/compare/v2.0.2...9b88afc9cd57fd75b655d5c71bd38146d07135fe )
Signed-off-by: dependabot[bot] <[email protected]
2021-04-26 14:08:40 +03:00
dependabot[bot]
8ac26b58ee
build(deps): bump codecov/codecov-action from v1.4.0 to v1.4.1 ( #5857 )
...
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action ) from v1.4.0 to v1.4.1.
- [Release notes](https://github.com/codecov/codecov-action/releases )
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md )
- [Commits](https://github.com/codecov/codecov-action/compare/v1.4.0...967e2b38a85a62bd61be5529ada27ebc109948c2 )
Signed-off-by: dependabot[bot] <[email protected]
2021-04-22 13:50:46 +03:00
Ruslan Kuprieiev
3472cb1a6a
tests: don't set sensitive env secrets for all steps ( #5823 )
...
Due to https://about.codecov.io/security-update/ , there is a potential that we've leaked these. This PR takes a more granular approach when setting secret env vars so that this doesn't happen in the future.
We'll change these secrets after this PR is merged.
2021-04-19 16:39:37 +03:00
dependabot[bot]
48890c22b7
build(deps): bump codecov/codecov-action from v1.3.2 to v1.4.0 ( #5838 )
...
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action ) from v1.3.2 to v1.4.0.
- [Release notes](https://github.com/codecov/codecov-action/releases )
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md )
- [Commits](https://github.com/codecov/codecov-action/compare/v1.3.2...0e28ff86a50029a44d10df6ed4c308711925a6a8 )
Signed-off-by: dependabot[bot] <[email protected]
2021-04-17 11:20:46 +05:45
dependabot[bot]
2472c04873
build(deps): bump actions/cache from v2.1.4 to v2.1.5 ( #5805 )
...
Bumps [actions/cache](https://github.com/actions/cache ) from v2.1.4 to v2.1.5.
- [Release notes](https://github.com/actions/cache/releases )
- [Commits](https://github.com/actions/cache/compare/v2.1.4...1a9e2138d905efd099035b49d8b7a3888c653ca8 )
Signed-off-by: dependabot[bot] <[email protected]
2021-04-13 14:23:05 +03:00
dependabot[bot]
c71a576a7a
build(deps): bump actions/setup-python from v2.2.1 to v2.2.2 ( #5804 )
...
Bumps [actions/setup-python](https://github.com/actions/setup-python ) from v2.2.1 to v2.2.2.
- [Release notes](https://github.com/actions/setup-python/releases )
- [Commits](https://github.com/actions/setup-python/compare/v2.2.1...dc73133d4da04e56a135ae2246682783cc7c7cb6 )
Signed-off-by: dependabot[bot] <[email protected]
2021-04-13 14:22:57 +03:00
dependabot[bot]
02d9684c33
build(deps): bump codecov/codecov-action from v1.3.1 to v1.3.2 ( #5770 )
...
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action ) from v1.3.1 to v1.3.2.
- [Release notes](https://github.com/codecov/codecov-action/releases )
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md )
- [Commits](https://github.com/codecov/codecov-action/compare/v1.3.1...9b0b9bbe2c64e9ed41413180dd7398450dfeee14 )
Signed-off-by: dependabot[bot] <[email protected]
2021-04-07 19:14:40 +05:45
dependabot[bot] and Saugat Pachhai
9e7e01ce9f
build(deps): bump pre-commit/action from v2.0.0 to v2.0.2 ( #5767 )
...
* build(deps): bump pre-commit/action from v2.0.0 to v2.0.1
Bumps [pre-commit/action](https://github.com/pre-commit/action ) from v2.0.0 to v2.0.1.
- [Release notes](https://github.com/pre-commit/action/releases )
- [Commits](https://github.com/pre-commit/action/compare/v2.0.0...b54880c283dfc80a3fc1baf61807e8ae84471361 )
Signed-off-by: dependabot[bot] <[email protected] [email protected]
2021-04-05 22:58:34 +05:45
Saugat Pachhai
cd9098211f
remove pytest-tap ( #5713 )
2021-03-26 20:43:57 +05:45
dependabot[bot]
c3bf0e1cb5
build(deps): bump actions/setup-python from v2.1.4 to v2.2.1 ( #5702 )
...
Bumps [actions/setup-python](https://github.com/actions/setup-python ) from v2.1.4 to v2.2.1.
- [Release notes](https://github.com/actions/setup-python/releases )
- [Commits](https://github.com/actions/setup-python/compare/v2.1.4...3105fb18c05ddd93efea5f9e0bef7a03a6e9e7df )
Signed-off-by: dependabot[bot] <[email protected]
2021-03-26 18:40:55 +05:45
dependabot[bot]
f1375fd706
build(deps): bump actions/cache from v2 to v2.1.4 ( #5699 )
...
Bumps [actions/cache](https://github.com/actions/cache ) from v2 to v2.1.4.
- [Release notes](https://github.com/actions/cache/releases )
- [Commits](https://github.com/actions/cache/compare/v2...26968a09c0ea4f3e233fdddbafd1166051a095f6 )
Signed-off-by: dependabot[bot] <[email protected]
2021-03-26 12:40:45 +00:00
dependabot[bot]
d04aaf4ae1
build(deps): bump actions/checkout from v2.3.2 to v2.3.4 ( #5698 )
...
Bumps [actions/checkout](https://github.com/actions/checkout ) from v2.3.2 to v2.3.4.
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](https://github.com/actions/checkout/compare/v2.3.2...5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f )
Signed-off-by: dependabot[bot] <[email protected]
2021-03-26 12:40:26 +00:00
dependabot[bot]
191b8c1a6d
build(deps): bump codecov/codecov-action from v1.0.13 to v1.3.1 ( #5701 )
...
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action ) from v1.0.13 to v1.3.1.
- [Release notes](https://github.com/codecov/codecov-action/releases )
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md )
- [Commits](https://github.com/codecov/codecov-action/compare/v1.0.13...fcebab03f26c7530a22baa63f06b3e0515f0c7cd )
Signed-off-by: dependabot[bot] <[email protected]
2021-03-26 12:40:17 +00:00
Saugat Pachhai
8c90771388
Add dependabot to update github-actions and python requirements ( #5697 )
2021-03-26 18:20:10 +05:45
Saugat Pachhai
6c00fedb46
remove flaky-service
...
Looks like the project is dead: https://github.com/GoogleCloudPlatform/flaky-service .
And, the site https://flaky.dev seems to be dead.
2021-03-11 15:21:12 +05:45
Batuhan Taskaya
9ae8bcdf52
gdrive: support service auth with json ( #5485 )
...
* gdrive: support service auth with json
* retry logic
2021-02-18 17:10:35 +02:00
Ruslan Kuprieiev
282e83c481
tests: temporarily exclude gdrivefs from extras ( #5483 )
...
When trying to publish to pypi, we get:
```
Invalid value for requires_dist. Error: Can't have direct dependency:
"gdrivefs @ git+https://github.com/intake/gdrivefs.git ; extra ==
'tests'"
```
2021-02-17 05:49:34 +02:00
Ruslan Kuprieiev
c635493104
tests: don't limit the number of pytest workers ( #5441 )
...
We used to do this because of gitpython leaks.
2021-02-11 12:27:00 +02:00
Ruslan Kuprieiev
4b938c5104
gha: temporarily disable macos 11
...
Those workers take a very long time to start up.
2021-02-06 03:07:17 +02:00
Batuhan Taskaya
adedd19c52
gha: switch to pre-commit/action for caching ( #5389 )
2021-02-03 17:14:46 +02:00
Batuhan Taskaya
b337396144
cache pip directory on github actions ( #5365 )
2021-02-01 13:48:03 +01:00
Saugat Pachhai
e33274723d
Move flags from tests to config ( #4943 )
2020-11-23 16:21:24 +05:45
Saugat Pachhai
806f8032d9
ci: reduce no. of MacOS jobs ( #4940 )
...
This was increasing the wait times of the CI.
Also enabled codecov failure to not fail the whole CI pipeline.
2020-11-23 12:12:58 +05:45
Saugat Pachhai
5faae2a6e9
test: allow selectively running tests for remotes ( #4920 )
...
Some of the tests are run by default, other requires
`--enable-{remote}` to run. The remotes that are run by default
also have a `--disable-{remote}` flag. All of the remotes can be
enabled using `--all` flag.
To use this functionality in the fixture,
`test_config` fixture has to be used, and requires to call
`test_config.requires(remote)`.
In an individual test, you can simply use `pytest.mark.remote`
to make them skip or run.
2020-11-23 11:43:39 +05:45
Ruslan Kuprieiev
9741f6d706
gha: rename "Tests build" -> "Tests"
2020-11-18 02:55:24 +02:00
Ruslan Kuprieiev
d5b7f7cf2c
gha: upgrade setup-python to 2.1.4
2020-11-17 05:50:34 +02:00
Ruslan Kuprieiev
1788fe95dd
gha: upgrade setup-python action to v2
2020-11-17 05:46:56 +02:00
Peter Rowlands (변기호)
96ead9425a
ci: add macos-11 @ py3.8, py3.9 to build matrix ( #4892 )
2020-11-16 15:11:24 +09:00
Saugat Pachhai
3ec64301b9
Send TAP result to flaky service only on master ( #4888 )
2020-11-12 20:05:52 +05:45
Saugat Pachhai
60b69b93cb
Add flaky-service ( #4834 )
...
Use master branch for flaky-service
Generate TAP results during pytest
2020-11-03 18:41:10 +05:45
Ruslan Kuprieiev
0736c64188
gha: don't create gs creds file if no creds
2020-10-27 00:23:34 +02:00
Ruslan Kuprieiev
bfecd7d8e8
gha: add gs creds ( #4791 )
...
* gha: add gs creds
Related to #4418
* tests: gs: don't use gcloud
2020-10-26 19:20:58 +02:00
Ruslan Kuprieiev
a5bdfa90f0
gha: add aws creds ( #4790 )
...
Related to #4418
2020-10-26 17:02:07 +02:00
Saugat Pachhai
ec05e6026f
Display coverage report on the GHA log and measure branch coverage ( #4760 )
...
* Display coverage report on the terminal and Measure branch coverage
* Add source path
* Install wheel, run in editable mode
2020-10-22 20:28:50 +03:00
Saugat Pachhai
e8c4924997
Make codecov push results on every run ( #4754 )
...
* Make codecov push results on every run
* codecov: do not comment on pull requests
2020-10-22 17:17:26 +03:00
Saugat Pachhai
Ruslan Kuprieiev
Peter Rowlands (변기호)
dependabot[bot]
GaoandRuslan Kuprieiev
Batuhan Taskaya