Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
9,393 Commits 7 Branches 579 Tags
a3b6722523c17784bf735b63ec3d294aff70ca85
Commit Graph
253 Commits
Author SHA1 Message Date
dependabot[bot] 13ca959e98 build(deps): Bump actions/cache from 2.1.6 to 2.1.7 
Bumps [actions/cache](https://github.com/actions/cache) from 2.1.6 to 2.1.7.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2.1.6...v2.1.7)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
 2021-11-24 08:47:13 +05:45
Saugat Pachhai (सौगात) 46289182d2 tests: run on Windows with Python 3.10 
pygit2 wheels should be now available.
Had to skip pytest-docker for Windows + Python 3.10 due to
a dependency issue with it's dependency, see:
https://github.com/docker/docker-py/issues/2902.
 2021-11-20 18:52:31 +02:00
Paweł Redzyński ab56e4bb9b gha: limit python version to 3.9.7 
Related: https://bugs.python.org/issue45235
Closes: #6956
 2021-11-11 01:14:03 +02:00
dependabot[bot] b927337a30 build(deps): bump actions/checkout from 2.3.5 to 2.4.0 
Bumps [actions/checkout](https://github.com/actions/checkout) from 2.3.5 to 2.4.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2.3.5...v2.4.0)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
 2021-11-03 12:47:22 +02:00
Saugat Pachhai 800fbc7ad0 Use -n=auto in tests 
Linux/Windows machines have only 2 cores, whereas macOS machines have 3 cores. Maybe we are stretching available resources by 2x, let's see if `-n=auto` improves test freezes.
 2021-10-22 08:46:55 +05:45
Peter Rowlands (변기호) 73f2fa8c93 git: use asyncssh as dulwich SSH vendor (#6797) 
* git: support client kwargs for auth in remote methods

* tests: use docker for git/ssh server

* move dulwich backend into subdir

* add base sync<->async support

* dulwich: add initial asyncssh SSH vendor

* dulwich: use asyncssh as the default SSH vendor

* add test for git over ssh

* test read (fetch) over ssh

* ci: re-enable ssh tests

temporary workaround for #6788, #6789

* move async stubs into dvc.scm.asyn

* tests: use sshfs and not dvc.fs.ssh

* allow explicit 0-length read in asyncssh vendor
 2021-10-21 14:36:45 +09:00
Saugat Pachhai 0c41a25bbe move some configs to pyproject.toml (#6834) 2021-10-19 19:47:51 +03:00
dependabot[bot] 4ed4766692 build(deps): Bump actions/checkout from 2.3.4 to 2.3.5 (#6817) 
Bumps [actions/checkout](https://github.com/actions/checkout) from 2.3.4 to 2.3.5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2.3.4...v2.3.5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-18 19:43:41 +05:45
Ruslan Kuprieiev e9a7e0657d dvc: drop 3.6 support (#6792) 
* dvc: drop 3.6 support

Considering that 3.6 has ~2 months until EOL, and our deps are
dropping support for it already (e.g. pygit2), we are droppipping
it as well. This will reduce the load on our test CI as well, since
we've started supporting 3.10 recently.

* setup.cfg: adjust requirements

* daemon: remove 3.6 logic

* tests: utils: remove pre 3.7 logic
 2021-10-12 18:53:28 +03:00
Saugat PachhaiandRuslan Kuprieiev 8abda93e3e use setuptools_scm and build_requires (#6775) 
* use setuptools_scm and build_requires

* remove kwargs from setup.py

* fix tests; fetch in depth in github actions

* use build to build sdist/wheel

* try updating pip and setuptools

* update pip

* remove or unpin wheel

* bring back the version in pyinstaller

* Update scripts/build.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2021-10-12 17:58:14 +03:00
Ruslan Kuprieiev 53255c22b3 gha: tests: temporarily disable some remote tests (#6789) 
* gha: tests: temporarily disable some remote tests

See #6788

* machine: upgrade tpi version

```
dvc/machine/backend/terraform.py:28:19: E1101: Instance of 'TerraformBackend' has no '_default_resource' member; maybe 'default_resource'? (no-member)
```

* tests: remotes: require --s3 for s3 tests
 2021-10-12 17:26:47 +03:00
Paweł Redzyński 8580e66d60 gha: move cancelaltion into other workflows (#6771) 2021-10-08 18:03:07 +03:00
Saugat Pachhai f628f670c9 advertise Python 3.10 support (#6772) 
* advertise Python 3.10 support

Pygit2 does not have windows wheels yet and no pyarrow
for 3.10. But these platform-dependent things and
remote-dependent things should not stop us from
advertising support as we are pure-python based
package and we cannot claim support for all of our
dependencies.

* set pygit2 for Python 3.6 to <1.7
 2021-10-08 15:50:05 +03:00
Saugat Pachhai c5415342d0 move requirements to setup.cfg (#6758) 
* move requirements to setup.cfg

* hash based on setup.cfg

* do not break webhdfs installation
 2021-10-08 07:00:14 +05:45
Paweł Redzyński dd86076ba6 gha: cancel previous workflows (#6756) 2021-10-07 17:59:44 +03:00
Ruslan Kuprieiev 32f435c47f gha: benchmarks: fix pytest flags (#6749) 
Kudos @skshetry for catching.
 2021-10-06 17:07:41 +03:00
Ruslan Kuprieiev 7b68a6f0ca gha: run benchmarks on PRs (#6733) 2021-10-05 14:18:43 +03:00
Saugat Pachhai ee823a59b9 Run on Python3.10 (#6745) 2021-10-05 16:46:53 +05:45
Saugat Pachhai 46ad8745d3 Use latest pygit2 from master for 3.10 (#6723) 
* Use latest pygit2 from master for 3.10

* ci: update brew
 2021-10-01 11:48:11 +05:45
Ruslan Kuprieiev e81c88e501 gha: increase timeouts (#6645) 2021-09-19 23:47:15 +03:00
Saugat Pachhai e049c4bab1 Set timeout for tests back to 30 
Reverts 93a200c
 2021-09-15 19:10:52 +05:45
Ruslan Kuprieiev 93a200cf5f gha: increase timeout 2021-09-14 18:37:10 +03:00
dependabot[bot] 1a1294848b build(deps): Bump codecov/codecov-action from 2.0.3 to 2.1.0 (#6606) 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2.0.3 to 2.1.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.3...v2.1.0)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-14 08:46:38 +05:45
Saugat Pachhai 1744028e09 Allow running package build manually on other PRs (#6553) 2021-09-07 22:28:50 +05:45
Saugat Pachhai f9bb38b96f Set timeout for jobs in Github Actions (#6545) 
* Set timeout for jobs in Github Actions

Sometimes tests are getting frozen(most likely an issue with `pytest-xdist`), and sometimes the pip resolver is slow due to some weird constraints set by us or one of our dependencies.

The default timeout is 360 minutes (6hrs), but our test completes on ~18 minutes at maximum, so I think 30mins should be enough to account for occasional CI slowdown.

Also keeping the total job timeout at 45 mins, we usually require just 4 mins of extra time to do installation and setup but giving it 11 more minutes just in case.

Similarly, `lint` completes in <5 mins usually, but giving it a 10 mins timeout.

This should give us early feedback on the issue.

* Update setup.cfg

* Update setup.cfg
 2021-09-07 14:08:22 +05:45
Ruslan Kuprieiev 6287b13ba3 gha: allow running tests on other branches (#6520) 
Simplifies development for all contributors.
 2021-09-02 17:55:54 +03:00
Peter Rowlands (변기호) a69edcc2c8 tpi: lazily import python_terraform (#6526) 
* machine: lazy load tpi/python_terraform only when needed

* remove terraform from package installers

* update tpi test imports
 2021-09-02 17:09:31 +03:00
Ruslan Kuprieiev 5d7476a539 gha: packages: don't forget terraform 2021-08-27 23:43:28 +03:00
Ruslan Kuprieiev 341ab4fe9b gha: packages: don't forget terraform 2021-08-27 23:38:21 +03:00
Peter Rowlands (변기호) 4ee98299ee dvc machine: implement initial terraform support (#6462) 
* tpi: add naive terraform-provider-iterative wrapper

* executor: add naive backend/terraform wrapper

* setup: add dvc[terraform] install extras

* config: add feature.executor feature flag

* fix named executor config sections

* executor: add poc add/modify/remove/init/destroy commands

* relax python-terraform requirement

* ci: install terraform extras

* tpi: support .tf.json format

* s/executor/machine

* terraform: use separate dirs per machine

* s/init/create/

* reserve local machine name

* add initial ssh skeleton

* add initial machine unit tests

* add initial machine command tests

* add tests for add/remove

* enable feature by default in test env

* close temp pemfile object so it can be used on windows

* cleanup pemfile syntax
 2021-08-27 12:46:24 +03:00
dependabot[bot] ceb0964fcf build(deps): Bump codecov/codecov-action from 2.0.2 to 2.0.3 (#6487) 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.2...v2.0.3)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-25 08:22:52 +05:45
Saugat Pachhai 1bb7f39e14 add support for Python 3.10 (#6422) 
* add support for Python 3.10

pyarrow is not available, so we are skipping tests for it.
And, restricting `hdfs` extras to be not installed in Python3.10.

Pathlib removed `_init` method and `init` parameter, so just working
around that. It's not a big change.

* skip windows on 3.10; update pip/wheel/setuptools during install

pywin32 is not yet available for 3.10 Python, so skipping those for now

* install libgit2 with Python3.10 as pygit2 wheel is not available

* install libgit2

* only run on macOS, libgit2-1.1 not available on ubuntu
 2021-08-16 16:24:28 +05:45
dependabot[bot] b294bca383 build(deps): bump codecov/codecov-action from 2.0.1 to 2.0.2 (#6361) 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.1...v2.0.2)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-28 10:21:10 +03:00
dependabot[bot] 005bca1da6 build(deps): bump codecov/codecov-action from 1.5.2 to 2.0.1 (#6334) 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1.5.2 to 2.0.1.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1.5.2...v2.0.1)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-20 14:06:50 +03:00
GaoandRuslan Kuprieiev 93f36d9351 oss: migrate to ossfs (#6307) 
* Migrate to ossfs

fix 6177

* Add verification to object stored on oss

* Change the oss remote's dependency.

pass all of the tests

* Performance optimaztion and ossfs version update

* Remove the argument to set the resumable get/put

1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default

* Use a new api for performance optimization

* Without filesize progress bar version

* With progress bar version

* Bump and add comments

1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger

* Change the name of kwrags following parent class.

* Bump to new ossfs and rewrite upload_fobj

Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj

* Optimization on rm_files

* Some optimization follow the old version.

* Update dvc/fs/oss.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/objects/db/oss.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* bump into ossfs 2021.7.2

* Bump into ossfs 2021.7.3OC

* Switch to real_oss_server

* Switch to a real oss server

* Use the oss bucket located in US.

* Remove some emulator related code

* Add envs to ci

* remove some oss pytest XFAIL

Because we didn't use emulator, we can remove some pytest XFAILs

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2021-07-13 15:27:34 +03:00
Saugat Pachhai b7c6df0ece Create CODEOWNERS (#6213) 2021-06-23 11:56:37 +05:45
dependabot[bot] 57c728269d build(deps): bump codecov/codecov-action from 1.5.0 to 1.5.2 (#6139) 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1.5.0 to 1.5.2.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1.5.0...v1.5.2)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-09 23:30:31 +05:45
dependabot[bot] bf5c28b71a build(deps): bump rtCamp/action-slack-notify from 2.1.3 to 2.2.0 (#6128) 
Bumps [rtCamp/action-slack-notify](https://github.com/rtCamp/action-slack-notify) from 2.1.3 to 2.2.0.
- [Release notes](https://github.com/rtCamp/action-slack-notify/releases)
- [Commits](https://github.com/rtCamp/action-slack-notify/compare/v2.1.3...v2.2.0)

---
updated-dependencies:
- dependency-name: rtCamp/action-slack-notify
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-07 14:04:32 +05:45
Batuhan Taskaya 0e3db0d323 ci: move to ubuntu-18 from ubuntu-16 (#6101) 2021-06-02 17:31:28 +03:00
dependabot[bot] efda015de4 build(deps): bump actions/cache from 2.1.5 to 2.1.6 (#6073) 
Bumps [actions/cache](https://github.com/actions/cache) from 2.1.5 to 2.1.6.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2.1.5...v2.1.6)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-05-28 14:43:23 +05:45
dependabot[bot] 0ca08d2fea build(deps): bump rtCamp/action-slack-notify from 2 to 2.1.3 (#5993) 
Bumps [rtCamp/action-slack-notify](https://github.com/rtCamp/action-slack-notify) from 2 to 2.1.3.
- [Release notes](https://github.com/rtCamp/action-slack-notify/releases)
- [Commits](https://github.com/rtCamp/action-slack-notify/compare/v2...v2.1.3)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-05-12 13:06:50 +05:45
dependabot[bot] 14e1511eee build(deps): bump actions/setup-ruby from 1 to 1.1.3 (#5994) 
Bumps [actions/setup-ruby](https://github.com/actions/setup-ruby) from 1 to 1.1.3.
- [Release notes](https://github.com/actions/setup-ruby/releases)
- [Commits](https://github.com/actions/setup-ruby/compare/v1...v1.1.3)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-05-12 13:06:40 +05:45
Saugat Pachhai eabf6ddf27 run gdrive tests in a single job, run all on schedule (#5974) 2021-05-06 17:09:34 +05:45
Saugat Pachhai c2f1a1e162 Notify CI failures to Slack (#5970) 2021-05-05 18:37:33 +05:45
dependabot[bot] c2734a2cd7 build(deps): bump codecov/codecov-action from v1.4.1 to v1.5.0 (#5957) 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from v1.4.1 to v1.5.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1.4.1...a1ed4b322b4b38cb846afb5a0ebfa17086917d27)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-05-04 17:45:01 +03:00
Ruslan Kuprieiev c9a748421a dependabot: add "maintenance" label (#5898) 2021-04-29 16:59:53 +03:00
dependabot[bot] af6921e9ac build(deps): bump pre-commit/action from v2.0.2 to v2.0.3 (#5856) 
Bumps [pre-commit/action](https://github.com/pre-commit/action) from v2.0.2 to v2.0.3.
- [Release notes](https://github.com/pre-commit/action/releases)
- [Commits](https://github.com/pre-commit/action/compare/v2.0.2...9b88afc9cd57fd75b655d5c71bd38146d07135fe)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-04-26 14:08:40 +03:00
dependabot[bot] 8ac26b58ee build(deps): bump codecov/codecov-action from v1.4.0 to v1.4.1 (#5857) 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from v1.4.0 to v1.4.1.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1.4.0...967e2b38a85a62bd61be5529ada27ebc109948c2)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-04-22 13:50:46 +03:00
Ruslan Kuprieiev 3472cb1a6a tests: don't set sensitive env secrets for all steps (#5823) 
Due to https://about.codecov.io/security-update/ , there is a potential that we've leaked these. This PR takes a more granular approach when setting secret env vars so that this doesn't happen in the future.

We'll change these secrets after this PR is merged.
 2021-04-19 16:39:37 +03:00
dependabot[bot] 48890c22b7 build(deps): bump codecov/codecov-action from v1.3.2 to v1.4.0 (#5838) 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from v1.3.2 to v1.4.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1.3.2...0e28ff86a50029a44d10df6ed4c308711925a6a8)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-04-17 11:20:46 +05:45