Support for WebHDFS (#4747)

* Add support for WebHDFS

* Style and linting fixes

* Implement abstract methods and disable no-member pylint errors

* Add init unit test

* Yield client and fix redefinition of hdfs

* Get dynamic hdfs Docker port

* Add support for open in WebHDFSTree

* Remove hdfscli_config fallbacks in WebHDFSTree

* Use port 50070 for webhdfs

* WebHDFSTree supports http URL and proper walking

* Fix failing unit test after refactoring

* Add overwrite argument to upload

* Add remaining func tests for webhdfs

* Remove print statements

* Allow dynamic attributes in pylint for hdfs library

* Comment error on instantiating HDFS client from config

* Remove depth argument from call to walk

* Update dvc/tree/webhdfs.py

* Remove unnecessary deque

* Check exception messages hdfs client failure from config

* Return client directly

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Split long line

* Add size to HashInfo

* Implement move via hdfs.rename

* Add missing line

* Make parent directories before moving file in WebHDFS

* Add total to progress bar

Co-authored-by: Ruslan Kuprieiev <[email protected]>

setup.py

@@ -98,13 +98,14 @@ ssh = ["paramiko[invoke]>=2.7.0"]
 
 # Remove the env marker if/when pyarrow is available for Python3.9
 hdfs = ["pyarrow>=2.0.0;  python_version < '3.9'"]
+webhdfs = ["hdfs==2.5.8"]
 webdav = ["webdavclient3>=3.14.5"]
 # gssapi should not be included in all_remotes, because it doesn't have wheels
 # for linux and mac, so it will fail to compile if user doesn't have all the
 # requirements, including kerberos itself. Once all the wheels are available,
 # we can start shipping it by default.
 ssh_gssapi = ["paramiko[invoke,gssapi]>=2.7.0"]
-all_remotes = gs + s3 + azure + ssh + oss + gdrive + hdfs + webdav
+all_remotes = gs + s3 + azure + ssh + oss + gdrive + hdfs + webhdfs + webdav
 
 # Extra dependecies to run tests
 tests_requirements = [
@@ -173,6 +174,7 @@ setup(
         "ssh": ssh,
         "ssh_gssapi": ssh_gssapi,
         "hdfs": hdfs,
+        "webhdfs": webhdfs,
         "webdav": webdav,
         "tests": tests_requirements,
     },