* remote.http: support _upload() via HTTP POST
- uploaded files are sent as chunked encoding POST data
Fixes#3247
* remote.http: add support for HTTP basic and digest auth
- username, password, ask_password options work the same way as for
ssh remotes
- basic_auth (bool) and digest_auth (bool) options added for http(s)
remotes
- digest_auth takes precedence over basic_auth if both are enabled
* remote.http: auth_method() unit tests
* tests: Add functional tests for HTTP remotes
- HTTP remotes now tested locally using a SimpleHTTPServer instance that
allows reading/writing to a temp directory
* tests: fix http tests on py3.5, 3.6
* remote.http: add custom auth method
* s/custom_header/custom_auth_header/
* Apply suggestions from code review
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
* master:
address @efiop's suggestions
s3: add support for top level empty directories
remote: http: calculate length basing on response and not head call
test: remote: http: clear test file name
HTTPError: reformat error message
Update dvc/remote/http.py
remote: http: rename http error, refactor test
remote: http: raise exception when download response with error status code
Support GDrive as remote
setup: don't forget contextlib2
remote: refactor ssh ask password code
remote: protect all remote client/session creation code with locks
test: refactor & remove redundant test fixtures
NoRemoteInExternalRepoError: dont pass cause of exception
perf: optimize cache listing for local, ssh and hdfs
remote: small .save_info()/.get_checksum() cleanup
Default chunk size for gs is 100M, which might be too much for
particular network. So if we are getting ConnectionError, we
should try lowering the chunk size until we reach the minimum
allowed chunk size of 256K. Also note that `chunk_size` must be a
multiple of 256K per the API specification.
Fixes#2572
* s3: list_objects -> list_objects_api
* s3: use path_info for _list_paths
* s3: separate _list_paths from _list_objects
* s3: support max items for listing objects/paths
* s3: take into account directories on `exists`
* s3: overwrite `isdir` method
* s3: custom collect directory entry for s3
* s3: strip checksum when collecting directories
* remote: change walk implementation to walk_files
* s3: add comment about isdir logic
* s3: use `dir_path` correctly on `isdir`
* tests: add a few unit tests for S3
* s3: use path_info join instead of posixpath
* tests: use parametrize for s3 tests
* remote: use posixpath accordingly
* tests: remove undeeded AWS variables from S3 test
* tests: rearrange parametrize values for readability
* tests: add encoding, use a single s3 instance, for loop instead of parameterize
* py2: fix encoding
* dvc: dedup used cache
This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts
* dvc: delay checksum names concatenation
Alexander Schepanovski
Guro Bokum
pawel
Max Risuhin
Aman Sharma
Mr. Outis
Veselin Borisov
Vera Sativa
VeraandRuslan Kuprieiev
Casper da Costa-Luis
Ruslan Kuprieiev
GeorgeSabu-MEC