Commit Graph
9 Commits
Author SHA1 Message Date
Batuhan Taskaya 597f730e7d s3: respect credentialpath/configpath (#5973) 2021-05-06 12:27:29 +03:00
Batuhan Taskaya a234f53b76 azure: use a custom event loop for authentication (#5958)
* azure: use a custom event loop for authentication

* add test_temp_event_loop

* wrap open() too!

* ignore aiohttp warnings for normal level use-cases

* rename tests/unit/remote/test_azure to tests/unit/fs/test_azure
2021-05-05 15:41:55 +03:00
Batuhan Taskaya d4a3df73df s3: fsspec migration (#5683)
* s3: migrate to s3fs

* s3: strip etag

* s3: disable instance cache

* tests: use the proper remote:// form in test_checkout

* tests: move/drop some tests

* setup: depend on s3fs

* tests: normalize s3 tests with some cleanups

* Use _download/_upload with old implementation

* fsspec: cache .exists() results

* setup: pin the boto version

* setup: install s3fs[boto3]

* s3: proper exception management

* s3: config improvements

* setup: reference without quotes

* setup: install boto separately

* setup: install moto server

* s3: mocking adjustments for moto_server

* s3: mark some fixtures with scope=session

* s3: s3_fake_creds_file scopes to func

* fsspec: add some docs regarding exists() optimization

* fsspec: exists() fallback

* fsspec: revert exists() optimization

* tests: ssl_verify => verify

* setup: temporarily fetch from master

* s3: add boto as a requirement

* tests: do an extra check regarding race conditions

* tests: use motoserver docker image

* tests: wrap config deletion with try/finally

* tests: disable s3 by default

* setup: upgrade fsspec
2021-05-05 11:36:10 +03:00
Saugat Pachhai 0206812d1b Fix test assertion with environment when using pyenv (#5765) 2021-04-05 22:56:05 +05:45
c3a61ad9cf Added SSL verification option (#5732)
* Added SSL verification option

* Changed param name to avoid conflicts

* verify_ssl now accepts strings

* Added simple unit tests

* Changed parameter name fron verify_ssl to ssl_verify so it's consistent with HTTP remotes

Forget to change 1 test

* add processing on ssl_verify param

typo

* revert back to bool param

* Update dvc/config_schema.py

Co-authored-by: Nigel <[email protected]>
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-03-31 23:31:38 +03:00
Ruslan Kuprieiev 4af3b3d20d dvc: refactor objects (#5607)
* dvc: get rid of DirInfo

DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.

As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.

* objects: move Tree into tree.py
2021-03-12 01:10:34 +02:00
Ruslan Kuprieiev 8f568ab197 state: use diskcache (#5505)
* state: only load when needed

* state: don't use separate load/dump

* checkout: group link removal logic together

* state: use diskcache.Cache

* staging: use pbar and save to state

* state: don't pass the whole repo to __init__

* staging: explicitly pass state object

* fs: get rid of odb property

* state: don't close the connection to cache

* repo: close: close state
2021-02-23 20:51:34 +02:00
Ruslan Kuprieiev 53ba51b5fe objects: separate staging (#5470) 2021-02-16 02:21:54 +02:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00