s3: fsspec migration (#5683)

* s3: migrate to s3fs

* s3: strip etag

* s3: disable instance cache

* tests: use the proper remote:// form in test_checkout

* tests: move/drop some tests

* setup: depend on s3fs

* tests: normalize s3 tests with some cleanups

* Use _download/_upload with old implementation

* fsspec: cache .exists() results

* setup: pin the boto version

* setup: install s3fs[boto3]

* s3: proper exception management

* s3: config improvements

* setup: reference without quotes

* setup: install boto separately

* setup: install moto server

* s3: mocking adjustments for moto_server

* s3: mark some fixtures with scope=session

* s3: s3_fake_creds_file scopes to func

* fsspec: add some docs regarding exists() optimization

* fsspec: exists() fallback

* fsspec: revert exists() optimization

* tests: ssl_verify => verify

* setup: temporarily fetch from master

* s3: add boto as a requirement

* tests: do an extra check regarding race conditions

* tests: use motoserver docker image

* tests: wrap config deletion with try/finally

* tests: disable s3 by default

* setup: upgrade fsspec
This commit is contained in:
Batuhan Taskaya
2021-05-05 11:36:10 +03:00
committed by GitHub
parent 256cb06921
commit d4a3df73df
14 changed files with 314 additions and 688 deletions
+4
View File
@@ -6,6 +6,10 @@ services:
command: azurite -L -l /data --blobHost 0.0.0.0 --queueHost 0.0.0.0
ports:
- "10000"
motoserver:
image: motoserver/moto
ports:
- "5000"
oss:
image: rkuprieiev/oss-emulator
ports: