* dvc: drop 3.6 support
Considering that 3.6 has ~2 months until EOL, and our deps are
dropping support for it already (e.g. pygit2), we are droppipping
it as well. This will reduce the load on our test CI as well, since
we've started supporting 3.10 recently.
* setup.cfg: adjust requirements
* daemon: remove 3.6 logic
* tests: utils: remove pre 3.7 logic
* use setuptools_scm and build_requires
* remove kwargs from setup.py
* fix tests; fetch in depth in github actions
* use build to build sdist/wheel
* try updating pip and setuptools
* update pip
* remove or unpin wheel
* bring back the version in pyinstaller
* Update scripts/build.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
1. update diff.py,experiments.py,metrics.py,params.py,status.py to
print prettify json
2. rewrite write_json in __init__.py to accept default function
3. add new unit test for write_json in tests/unit/ui/test_console.py
4. Fix#6712
Co-authored-by: Zhenyu Gao <[email protected]>
This was used before on opening http urls as a stream,
but we no longer use those (as we use fsspec's open method).
One thing to note however is that, compared to previous
implementation, fsspec's http filesystem's open is not very
robust to failures.
```
2021-09-21T00:15:31.6789059Z ____________________ ERROR at setup of test_get_url[azure] _____________________
2021-09-21T00:15:31.6790443Z [gw0] linux -- Python 3.7.12 /opt/hostedtoolcache/Python/3.7.12/x64/bin/python
2021-09-21T00:15:31.6791202Z
2021-09-21T00:15:31.6792291Z request = <FixtureRequest for <Function test_get_url[azure]>>
2021-09-21T00:15:31.6793561Z
2021-09-21T00:15:31.6794120Z def fill(request):
2021-09-21T00:15:31.6794792Z item = request._pyfuncitem
2021-09-21T00:15:31.6795627Z fixturenames = getattr(item, "fixturenames", None)
2021-09-21T00:15:31.6797090Z if fixturenames is None:
2021-09-21T00:15:31.6797922Z fixturenames = request.fixturenames
2021-09-21T00:15:31.6798497Z
2021-09-21T00:15:31.6799393Z if hasattr(item, 'callspec'):
2021-09-21T00:15:31.6800256Z for param, val in sorted_by_dependency(item.callspec.params, fixturenames):
2021-09-21T00:15:31.6801309Z if val is not None and is_lazy_fixture(val):
2021-09-21T00:15:31.6802233Z > item.callspec.params[param] = request.getfixturevalue(val.name)
2021-09-21T00:15:31.6802912Z
2021-09-21T00:15:31.6803937Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pytest_lazyfixture.py:35:
2021-09-21T00:15:31.6804952Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-09-21T00:15:31.6806085Z /home/runner/work/dvc/dvc/tests/remotes/azure.py:102: in azure
2021-09-21T00:15:31.6807090Z container.get_container_properties()
2021-09-21T00:15:31.6808748Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/core/tracing/decorator.py:83: in wrapper_use_tracer
2021-09-21T00:15:31.6822551Z return func(*args, **kwargs)
2021-09-21T00:15:31.6824143Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/storage/blob/_container_client.py:497: in get_container_properties
2021-09-21T00:15:31.6825038Z process_storage_error(error)
2021-09-21T00:15:31.6825652Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-09-21T00:15:31.6825908Z
2021-09-21T00:15:31.6826916Z storage_error = HttpResponseError("Operation returned an invalid status 'The value for one of the HTTP headers is not in the correct format.'")
2021-09-21T00:15:31.6827586Z
2021-09-21T00:15:31.6828351Z def process_storage_error(storage_error): # pylint:disable=too-many-statements
2021-09-21T00:15:31.6829086Z raise_error = HttpResponseError
2021-09-21T00:15:31.6829597Z serialized = False
2021-09-21T00:15:31.6830066Z if not storage_error.response:
2021-09-21T00:15:31.6830554Z raise storage_error
2021-09-21T00:15:31.6831558Z # If it is one of those three then it has been serialized prior by the generated layer.
2021-09-21T00:15:31.6832480Z if isinstance(storage_error, (PartialBatchErrorException,
2021-09-21T00:15:31.6833636Z ClientAuthenticationError, ResourceNotFoundError, ResourceExistsError)):
2021-09-21T00:15:31.6834687Z serialized = True
2021-09-21T00:15:31.6835753Z error_code = storage_error.response.headers.get('x-ms-error-code')
2021-09-21T00:15:31.6836524Z error_message = storage_error.message
2021-09-21T00:15:31.6837001Z additional_data = {}
2021-09-21T00:15:31.6837398Z error_dict = {}
2021-09-21T00:15:31.6837739Z try:
2021-09-21T00:15:31.6838506Z error_body = ContentDecodePolicy.deserialize_from_http_generics(storage_error.response)
2021-09-21T00:15:31.6839345Z # If it is an XML response
2021-09-21T00:15:31.6839837Z if isinstance(error_body, Element):
2021-09-21T00:15:31.6840285Z error_dict = {
2021-09-21T00:15:31.6840750Z child.tag.lower(): child.text
2021-09-21T00:15:31.6841435Z for child in error_body
2021-09-21T00:15:31.6841822Z }
2021-09-21T00:15:31.6842195Z # If it is a JSON response
2021-09-21T00:15:31.6842692Z elif isinstance(error_body, dict):
2021-09-21T00:15:31.6843413Z error_dict = error_body.get('error', {})
2021-09-21T00:15:31.6843893Z elif not error_code:
2021-09-21T00:15:31.6844347Z _LOGGER.warning(
2021-09-21T00:15:31.6845611Z 'Unexpected return type % from ContentDecodePolicy.deserialize_from_http_generics.', type(error_body))
2021-09-21T00:15:31.6846887Z error_dict = {'message': str(error_body)}
2021-09-21T00:15:31.6847294Z
2021-09-21T00:15:31.6847730Z # If we extracted from a Json or XML response
2021-09-21T00:15:31.6848205Z if error_dict:
2021-09-21T00:15:31.6848792Z error_code = error_dict.get('code')
2021-09-21T00:15:31.6849704Z error_message = error_dict.get('message')
2021-09-21T00:15:31.6850578Z additional_data = {k: v for k, v in error_dict.items() if k not in {'code', 'message'}}
2021-09-21T00:15:31.6851392Z except DecodeError:
2021-09-21T00:15:31.6851801Z pass
2021-09-21T00:15:31.6852290Z
2021-09-21T00:15:31.6852601Z try:
2021-09-21T00:15:31.6853156Z # This check would be unnecessary if we have already serialized the error
2021-09-21T00:15:31.6853821Z if error_code and not serialized:
2021-09-21T00:15:31.6854423Z error_code = StorageErrorCode(error_code)
2021-09-21T00:15:31.6855164Z if error_code in [StorageErrorCode.condition_not_met,
2021-09-21T00:15:31.6856229Z StorageErrorCode.blob_overwritten]:
2021-09-21T00:15:31.6856938Z raise_error = ResourceModifiedError
2021-09-21T00:15:31.6857739Z if error_code in [StorageErrorCode.invalid_authentication_info,
2021-09-21T00:15:31.6858641Z StorageErrorCode.authentication_failed]:
2021-09-21T00:15:31.6859468Z raise_error = ClientAuthenticationError
2021-09-21T00:15:31.6860256Z if error_code in [StorageErrorCode.resource_not_found,
2021-09-21T00:15:31.6861256Z StorageErrorCode.cannot_verify_copy_source,
2021-09-21T00:15:31.6861998Z StorageErrorCode.blob_not_found,
2021-09-21T00:15:31.6862716Z StorageErrorCode.queue_not_found,
2021-09-21T00:15:31.6863718Z StorageErrorCode.container_not_found,
2021-09-21T00:15:31.6864526Z StorageErrorCode.parent_not_found,
2021-09-21T00:15:31.6865432Z StorageErrorCode.share_not_found]:
2021-09-21T00:15:31.6866549Z raise_error = ResourceNotFoundError
2021-09-21T00:15:31.6867609Z if error_code in [StorageErrorCode.account_already_exists,
2021-09-21T00:15:31.6868499Z StorageErrorCode.account_being_created,
2021-09-21T00:15:31.6869383Z StorageErrorCode.resource_already_exists,
2021-09-21T00:15:31.6870728Z StorageErrorCode.resource_type_mismatch,
2021-09-21T00:15:31.6871688Z StorageErrorCode.blob_already_exists,
2021-09-21T00:15:31.6872429Z StorageErrorCode.queue_already_exists,
2021-09-21T00:15:31.6873252Z StorageErrorCode.container_already_exists,
2021-09-21T00:15:31.6874124Z StorageErrorCode.container_being_deleted,
2021-09-21T00:15:31.6874922Z StorageErrorCode.queue_being_deleted,
2021-09-21T00:15:31.6875857Z StorageErrorCode.share_already_exists,
2021-09-21T00:15:31.6876608Z StorageErrorCode.share_being_deleted]:
2021-09-21T00:15:31.6877289Z raise_error = ResourceExistsError
2021-09-21T00:15:31.6877807Z except ValueError:
2021-09-21T00:15:31.6878259Z # Got an unknown error code
2021-09-21T00:15:31.6878648Z pass
2021-09-21T00:15:31.6878942Z
2021-09-21T00:15:31.6879424Z # Error message should include all the error properties
2021-09-21T00:15:31.6879908Z try:
2021-09-21T00:15:31.6880416Z error_message += "\nErrorCode:{}".format(error_code.value)
2021-09-21T00:15:31.6881289Z except AttributeError:
2021-09-21T00:15:31.6883082Z error_message += "\nErrorCode:{}".format(error_code)
2021-09-21T00:15:31.6883807Z for name, info in additional_data.items():
2021-09-21T00:15:31.6884446Z error_message += "\n{}:{}".format(name, info)
2021-09-21T00:15:31.6884932Z
2021-09-21T00:15:31.6885741Z # No need to create an instance if it has already been serialized by the generated layer
2021-09-21T00:15:31.6886538Z if serialized:
2021-09-21T00:15:31.6887019Z storage_error.message = error_message
2021-09-21T00:15:31.6887526Z error = storage_error
2021-09-21T00:15:31.6887908Z else:
2021-09-21T00:15:31.6888499Z error = raise_error(message=error_message, response=storage_error.response)
2021-09-21T00:15:31.6889380Z # Ensure these properties are stored in the error instance as well (not just the error message)
2021-09-21T00:15:31.6890078Z error.error_code = error_code
2021-09-21T00:15:31.6890630Z error.additional_info = additional_data
2021-09-21T00:15:31.6891956Z # error.args is what's surfaced on the traceback - show error message in all cases
2021-09-21T00:15:31.6892672Z error.args = (error.message,)
2021-09-21T00:15:31.6893112Z try:
2021-09-21T00:15:31.6893784Z # `from None` prevents us from double printing the exception (suppresses generated layer error context)
2021-09-21T00:15:31.6894873Z > exec("raise error from None") # pylint: disable=exec-used # nosec
2021-09-21T00:15:31.6896126Z E azure.core.exceptions.HttpResponseError: The value for one of the HTTP headers is not in the correct format.
2021-09-21T00:15:31.6897385Z E RequestId:d83b8102-bde1-4a77-aa7a-b67bbc2c4a80
2021-09-21T00:15:31.6898080Z E Time:2021-09-21T00:09:53.815Z
2021-09-21T00:15:31.6898578Z E ErrorCode:InvalidHeaderValue
2021-09-21T00:15:31.6899259Z E headername:x-ms-version
2021-09-21T00:15:31.6900066Z E headervalue:2020-10-02
2021-09-21T00:15:31.6900355Z
2021-09-21T00:15:31.6901781Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/storage/blob/_shared/response_handlers.py:177: HttpResponseError
```
* rebased PR 6111 onto master
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use make_tmp_dir fixture in tests
Co-authored-by: U119511 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Alexander Meier <[email protected]>
Co-authored-by: meierale <[email protected]>
* tests: additional callbacks test for remaining filesystems
* requirements: bump pydrive2 to 1.9.4
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* machine: add `machine list`&`machine modify`
1. add new command `dvc machine list`
2. complete `dvc machine modify`
3. add new unit tests for this two commands to ensure the call.
4. add functionality tests for two commands
* Update dvc/config_schema.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update tests/func/machine/test_machine_config.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some reviewed problems
1. capitalize global variables.
2. remove precision restriction.
3. solve windows path seperator problem.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* dvc cli: fixed error in 'dvc config --list' in non repo dir
Fixes#6496
* Issue 6496: Added test that covers the issue
* dvc cli: fixes error in 'dvc config <name>' in non dvc repo dir
Fixes the specified error. Did some refactoring based on previous commits.
Also added a test case for setting config in non dvc repo dir.
Fixes#6496
* Issue-6496: dvc config --<level> and dvc config --list --<level> will now throw config error if ran in non dvc repo with repo only level
* Issue 6496: Segregated config levels into repo and system levels that helped in simplifying the logic to decide appropriate level at the time of cli operation.
The current ci failure was caused by wrong order of the experiments.
Because our test script runs so fast that they might share the same
commit timestamp, that makes them in an uncertain order. And then gives
a wrong branch_base.
* output: use remote kwarg instead of odb
Fixes#6596
* Update dvc/output.py
Co-authored-by: Dave Berenbaum <[email protected]>
* add test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Dave Berenbaum <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fs.download: add callback support
* fix lint issues
* compat for gdrive, use fs to upload text for tests
* fix typing issues
* skip gdrive test
* Update tests/func/test_fs.py
* fs.upload: add support for optional callback
Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
David de la Iglesia Castro
Paweł Redzyński
Ruslan Kuprieiev
Michael Lingelbach
Batuhan Taskaya
dependabot[bot]andPaweł Redzyński
eggqq007andZhenyu Gao
Amey Kulkarni
Nikita Kodenko
Amey Kulkarni
