Commit Graph
25 Commits
Author SHA1 Message Date
pre-commit-ci[bot]andSaugat Pachhai 502b3df6c9 [pre-commit.ci] pre-commit autoupdate (#9756)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/asottile/pyupgrade: v3.9.0 → v3.10.1](https://github.com/asottile/pyupgrade/compare/v3.9.0...v3.10.1)
- [github.com/astral-sh/ruff-pre-commit: v0.0.278 → v0.0.281](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.278...v0.0.281)
- [github.com/PyCQA/flake8: 6.0.0 → 6.1.0](https://github.com/PyCQA/flake8/compare/6.0.0...6.1.0)

* fix issues reported by ruff

- also remove redundant flake8 plugins
- remove beautysh as we don't have any scripts now
- remove pyupgrade as it is redundant due to ruff

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-08-02 18:08:30 +05:45
Ruslan Kuprieiev b6f2a8e653 tests: remove redundant remote tests
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.

This PR also removes some redundant/broken remote tests.
2021-12-03 22:45:41 +02:00
Ruslan Kuprieiev 09f48a5aaa tests: introduce dvc.testing
This is a very early version of test utils for dvc plugins. This module
consists of existing tests with minimal changes and is going to evolve
a lot as our plugins are getting filled in and pytest-test-utils matures.
2021-12-02 09:08:40 +02:00
Peter Rowlands (변기호) 73f2fa8c93 git: use asyncssh as dulwich SSH vendor (#6797)
* git: support client kwargs for auth in remote methods

* tests: use docker for git/ssh server

* move dulwich backend into subdir

* add base sync<->async support

* dulwich: add initial asyncssh SSH vendor

* dulwich: use asyncssh as the default SSH vendor

* add test for git over ssh

* test read (fetch) over ssh

* ci: re-enable ssh tests

temporary workaround for #6788, #6789

* move async stubs into dvc.scm.asyn

* tests: use sshfs and not dvc.fs.ssh

* allow explicit 0-length read in asyncssh vendor
2021-10-21 14:36:45 +09:00
dependabot[bot]andPaweł Redzyński 5a4f4d3a8e build(deps): Bump filelock from 3.2.0 to 3.3.0 (#6737)
* build(deps): Bump filelock from 3.2.0 to 3.3.0

Bumps [filelock](https://github.com/tox-dev/py-filelock) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Commits](https://github.com/tox-dev/py-filelock/compare/3.2.0...3.3.0)

---
updated-dependencies:
- dependency-name: filelock
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* tests: remotes: ignore abstract class instantiation for filelock

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Paweł Redzyński <[email protected]>
2021-10-08 15:36:33 +05:45
Batuhan Taskaya 4f3b28e268 webhdfs: migrate to fsspec (#6662)
* webhdfs: migrate to fsspec

* skip the tests that can't use hdfs
2021-10-07 17:02:31 +03:00
Batuhan Taskaya 6d3045320b ssh: use a docker-based testing environment (#6677) 2021-09-23 16:00:48 +03:00
Ruslan Kuprieiev 4382b80d3b dvcfile: support remote per output (#6486)
Related to #2095
2021-08-27 12:45:25 +03:00
Saugat Pachhai b9000ed926 pre-commit: update black to 21.5b1 (#6032)
* pre-commit: update black to 21.5b1

also updated others, but black is more noticeable

* remove magic trailing comma
2021-05-18 15:51:15 +00:00
Batuhan Taskaya d4a3df73df s3: fsspec migration (#5683)
* s3: migrate to s3fs

* s3: strip etag

* s3: disable instance cache

* tests: use the proper remote:// form in test_checkout

* tests: move/drop some tests

* setup: depend on s3fs

* tests: normalize s3 tests with some cleanups

* Use _download/_upload with old implementation

* fsspec: cache .exists() results

* setup: pin the boto version

* setup: install s3fs[boto3]

* s3: proper exception management

* s3: config improvements

* setup: reference without quotes

* setup: install boto separately

* setup: install moto server

* s3: mocking adjustments for moto_server

* s3: mark some fixtures with scope=session

* s3: s3_fake_creds_file scopes to func

* fsspec: add some docs regarding exists() optimization

* fsspec: exists() fallback

* fsspec: revert exists() optimization

* tests: ssl_verify => verify

* setup: temporarily fetch from master

* s3: add boto as a requirement

* tests: do an extra check regarding race conditions

* tests: use motoserver docker image

* tests: wrap config deletion with try/finally

* tests: disable s3 by default

* setup: upgrade fsspec
2021-05-05 11:36:10 +03:00
Ruslan Kuprieiev e0842ce01e dvc: rename cache -> odb (#5481) 2021-02-16 22:49:25 +02:00
Ruslan Kuprieiev d038bca05e tests: add basic [web]hdfs mockers (#5279) 2021-01-19 20:03:16 +02:00
Saugat Pachhai dcba0e8da4 pre-commit: update hooks (#4964) 2020-11-25 17:57:32 +05:45
Tomás Farías SantanaandRuslan Kuprieiev ca2e6a004c Support for WebHDFS (#4747)
* Add support for WebHDFS

* Style and linting fixes

* Implement abstract methods and disable no-member pylint errors

* Add init unit test

* Yield client and fix redefinition of hdfs

* Get dynamic hdfs Docker port

* Add support for open in WebHDFSTree

* Remove hdfscli_config fallbacks in WebHDFSTree

* Use port 50070 for webhdfs

* WebHDFSTree supports http URL and proper walking

* Fix failing unit test after refactoring

* Add overwrite argument to upload

* Add remaining func tests for webhdfs

* Remove print statements

* Allow dynamic attributes in pylint for hdfs library

* Comment error on instantiating HDFS client from config

* Remove depth argument from call to walk

* Update dvc/tree/webhdfs.py

* Remove unnecessary deque

* Check exception messages hdfs client failure from config

* Return client directly

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Split long line

* Add size to HashInfo

* Implement move via hdfs.rename

* Add missing line

* Make parent directories before moving file in WebHDFS

* Add total to progress bar

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-11-17 20:02:02 +02:00
Saugat Pachhai 72eb4a5737 Add test server and tests for webdav (#4827)
* Add test server and tests for webdav

* Fix webdav unittests
2020-11-12 13:52:46 +05:45
Juliano Efson SalesandRuslan Kuprieiev 052b15a2a8 tests/remote: adding tests for real oss (#4735)
* tests/remote: adding tests for real oss

the new test suggests #4633 is a bug in the emulator.

* Update tests/func/test_data_cloud.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update tests/remotes/oss.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* tests: correcting small lint issue

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-10-18 16:21:51 +03:00
Paweł Redzyński b31bd1cb30 github actions tests: initial (#4469)
* github actions tests: initial

* build: tests: add codecov action
2020-09-22 23:05:06 +03:00
Saugat Pachhai 1cfccef22e update flake8 to latest version (#4403) 2020-08-17 14:26:10 +05:45
Ruslan Kuprieiev 102187be58 tests: check if docker is present (#4263)
https://travis-ci.com/github/iterative/dvc/jobs/364061489#L3623
2020-07-23 05:55:46 +03:00
Ruslan Kuprieiev 4c9245a3b3 tests: remove real SSH (#4200)
* tests: remove real SSH

No longer needed, all tests are using mocked ssh already.

* travis: don't setup ssh
2020-07-14 13:55:14 +03:00
Ruslan Kuprieiev 375998c664 tests: hdfs: use hadoop/docker fixtures (#4183)
Fixed #4054
2020-07-10 21:38:48 +03:00
Ruslan Kuprieiev 0899b277c0 tests: launch oss/azure containers from fixtures (#4178)
* tests: launch oss/azure containers from fixtures

Related to #4054

* ci: hdfs: remove unneeded fix

We are using pyarrow 0.17.0 already.
2020-07-08 19:54:17 +03:00
Ruslan Kuprieiev 30212f886b tests: create remote fixture (#4163)
Usually we need it to parametrize tests with, so there is no
reason to duplicate the code. I've left `local_remote` fixture
just because it is so convenient and often used in a lot of tests.
2020-07-03 22:30:56 +03:00
Ruslan Kuprieiev 07da9f1e5c tests: remotes: use TmpDir-like fixtures (#4140) 2020-07-02 04:46:40 +03:00
Ruslan Kuprieiev 51a8c782ae tests: create remote fixtures (#4063)
* tests: use remote fixtures

* tests: use remote fixtures for data cloud tests

* tests: treat remotes as tmp dirs
2020-06-18 01:36:13 +03:00