Commit Graph
13 Commits
Author SHA1 Message Date
Saugat Pachhai dcba0e8da4 pre-commit: update hooks (#4964) 2020-11-25 17:57:32 +05:45
Tomás Farías SantanaandRuslan Kuprieiev ca2e6a004c Support for WebHDFS (#4747)
* Add support for WebHDFS

* Style and linting fixes

* Implement abstract methods and disable no-member pylint errors

* Add init unit test

* Yield client and fix redefinition of hdfs

* Get dynamic hdfs Docker port

* Add support for open in WebHDFSTree

* Remove hdfscli_config fallbacks in WebHDFSTree

* Use port 50070 for webhdfs

* WebHDFSTree supports http URL and proper walking

* Fix failing unit test after refactoring

* Add overwrite argument to upload

* Add remaining func tests for webhdfs

* Remove print statements

* Allow dynamic attributes in pylint for hdfs library

* Comment error on instantiating HDFS client from config

* Remove depth argument from call to walk

* Update dvc/tree/webhdfs.py

* Remove unnecessary deque

* Check exception messages hdfs client failure from config

* Return client directly

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Split long line

* Add size to HashInfo

* Implement move via hdfs.rename

* Add missing line

* Make parent directories before moving file in WebHDFS

* Add total to progress bar

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-11-17 20:02:02 +02:00
Saugat Pachhai 72eb4a5737 Add test server and tests for webdav (#4827)
* Add test server and tests for webdav

* Fix webdav unittests
2020-11-12 13:52:46 +05:45
Juliano Efson SalesandRuslan Kuprieiev 052b15a2a8 tests/remote: adding tests for real oss (#4735)
* tests/remote: adding tests for real oss

the new test suggests #4633 is a bug in the emulator.

* Update tests/func/test_data_cloud.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update tests/remotes/oss.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* tests: correcting small lint issue

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-10-18 16:21:51 +03:00
Paweł Redzyński b31bd1cb30 github actions tests: initial (#4469)
* github actions tests: initial

* build: tests: add codecov action
2020-09-22 23:05:06 +03:00
Saugat Pachhai 1cfccef22e update flake8 to latest version (#4403) 2020-08-17 14:26:10 +05:45
Ruslan Kuprieiev 102187be58 tests: check if docker is present (#4263)
https://travis-ci.com/github/iterative/dvc/jobs/364061489#L3623
2020-07-23 05:55:46 +03:00
Ruslan Kuprieiev 4c9245a3b3 tests: remove real SSH (#4200)
* tests: remove real SSH

No longer needed, all tests are using mocked ssh already.

* travis: don't setup ssh
2020-07-14 13:55:14 +03:00
Ruslan Kuprieiev 375998c664 tests: hdfs: use hadoop/docker fixtures (#4183)
Fixed #4054
2020-07-10 21:38:48 +03:00
Ruslan Kuprieiev 0899b277c0 tests: launch oss/azure containers from fixtures (#4178)
* tests: launch oss/azure containers from fixtures

Related to #4054

* ci: hdfs: remove unneeded fix

We are using pyarrow 0.17.0 already.
2020-07-08 19:54:17 +03:00
Ruslan Kuprieiev 30212f886b tests: create remote fixture (#4163)
Usually we need it to parametrize tests with, so there is no
reason to duplicate the code. I've left `local_remote` fixture
just because it is so convenient and often used in a lot of tests.
2020-07-03 22:30:56 +03:00
Ruslan Kuprieiev 07da9f1e5c tests: remotes: use TmpDir-like fixtures (#4140) 2020-07-02 04:46:40 +03:00
Ruslan Kuprieiev 51a8c782ae tests: create remote fixtures (#4063)
* tests: use remote fixtures

* tests: use remote fixtures for data cloud tests

* tests: treat remotes as tmp dirs
2020-06-18 01:36:13 +03:00