* git: support client kwargs for auth in remote methods
* tests: use docker for git/ssh server
* move dulwich backend into subdir
* add base sync<->async support
* dulwich: add initial asyncssh SSH vendor
* dulwich: use asyncssh as the default SSH vendor
* add test for git over ssh
* test read (fetch) over ssh
* ci: re-enable ssh tests
temporary workaround for #6788, #6789
* move async stubs into dvc.scm.asyn
* tests: use sshfs and not dvc.fs.ssh
* allow explicit 0-length read in asyncssh vendor
* `dvc machine`: add a new command `rename` to it
fix#6481
1. add new command `dvc machine rename`
2. add a unit test for the call.
3. add several functional tests for it.t push myself fix6481
* Move rename logic to tpi
* use tpi.state_mv instead of rename
* Rebase to a newer version
* Update tpi version
* Remove unused assert and try catch
* throw error if new parameters are added by dvc run exp -S
Throws an error if dvc run exp --set-param references a parameter key path
that is not in the target config file. This is done to prevent silent errors
in experimentation when new params are accidentally added instead of modifying
current ones due to typos.
This implementation is based on initial suggestions in issue #5477.
* fix mypy confusion from overwriting a var
* tests: remove modify param test that was adding params
One of the test in test_modify_params was still adding a parameter
* tests: remove param modification during exp run in test_untracked
Spurious usage of modifying a parameter when the params.yaml is not yet existing in test_untracked.
* remove benedict dependency for verifying params
Instead, the keypath util module from benedict is adapted into the dvc codebased and used
natively to verify that a keypath exists within a dict.
* exp: list all invalid param modifications in error message
* exp-run: more specific exception for new params via --set-param
Uses established MissingPramsError instead of DvcException. Modified message to be more similar to other uses of this exception.
* Revert "remove benedict dependency for verifying params"
This reverts commit 6170f16dcc4033667fbacb0d0bc13c60898f58c4.
* exp: use internal benedict import for validating params
* exp: combine new param validation into merge_params
Moves new param validation into existing merge_params function as an optional check
* exp: minor ammendments to error message
* Revert "tests: remove param modification during exp run in test_untracked"
This reverts commit c9eb1a62af58586509ded362cc79e75ddc4b119c.
* exp: fix test_untracked to not fail due to new params
Fixed by creating the params file with the new parameter already present.
* machine: Add a new command `dvc machine status`
fix#6482
1. add a new command `dvc machien status`
2. add a unit test for the cli call.
3. add a unit test for the shown result.
* Update dvc/command/machine.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Update dvc/command/machine.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Update machine.py
* Remove sensitive fields
* Update dvc/machine/__init__.py
Co-authored-by: Casper da Costa-Luis <[email protected]>
* Update tests/func/machine/__init__.py
Co-authored-by: Casper da Costa-Luis <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use white list to limit shown fields
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Casper da Costa-Luis <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* dvc: drop 3.6 support
Considering that 3.6 has ~2 months until EOL, and our deps are
dropping support for it already (e.g. pygit2), we are droppipping
it as well. This will reduce the load on our test CI as well, since
we've started supporting 3.10 recently.
* setup.cfg: adjust requirements
* daemon: remove 3.6 logic
* tests: utils: remove pre 3.7 logic
* use setuptools_scm and build_requires
* remove kwargs from setup.py
* fix tests; fetch in depth in github actions
* use build to build sdist/wheel
* try updating pip and setuptools
* update pip
* remove or unpin wheel
* bring back the version in pyinstaller
* Update scripts/build.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* advertise Python 3.10 support
Pygit2 does not have windows wheels yet and no pyarrow
for 3.10. But these platform-dependent things and
remote-dependent things should not stop us from
advertising support as we are pure-python based
package and we cannot claim support for all of our
dependencies.
* set pygit2 for Python 3.6 to <1.7
Dependabot was throwing Illformed requirements `"< '3.10'"` for the
hdfs. But I noticed that it did accept other requirements where we
do similar. So it seems the indentation fixes it.
Even then, dependabot did not update any dependencies. I removed
setup.py file, after which immediately it started creating PRs.
So, I recknoed that it must have been due to some mechanism
in place from dependabot to disallow execution, so I tried to
remove few blocks of code from setup.py. And, I was eventually
able to figure out that it was from the block where we import version.
So, I added a try-except block and it magically again worked. :)
Let's see how it goes.
1. update diff.py,experiments.py,metrics.py,params.py,status.py to
print prettify json
2. rewrite write_json in __init__.py to accept default function
3. add new unit test for write_json in tests/unit/ui/test_console.py
4. Fix#6712
Co-authored-by: Zhenyu Gao <[email protected]>
* doctor: fixes unknown cache director error on some windows mapped network shares
On Windows, resolve calls os.path._getfinalpathname, which calls GetFinalPathNameByHandle.
This starts from the final resolved path of the file in the NT namespace (e.g. \Device\Mup\server\share...)
and works backward as required to get the VOLUME_NAME_DOS. If the final path is a UNC share (i.e. \Device\Mup),
We fix this by calling resolve() also on the partition mountpoints
We also take the opportunity to make checks in case a PermissionError is returned to us
Fixes github.com/iterative/dvc/issues/6305
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* doctor: fixes unknown cache director error on some windows mapped network shares
On Windows, resolve calls os.path._getfinalpathname, which calls GetFinalPathNameByHandle.
This starts from the final resolved path of the file in the NT namespace (e.g. \Device\Mup\server\share...)
and works backward as required to get the VOLUME_NAME_DOS. If the final path is a UNC share (i.e. \Device\Mup),
We fix this by calling resolve() also on the partition mountpoints
We also take the opportunity to make checks in case a PermissionError is returned to us
Fixes#6305
Co-authored-by: CHETOUAN <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: CHETOUAN <[email protected]>
* exp init: introduce `--with={default,live}` flag
This flag will select the different mode of outputs of stage.
With `--with=live`, it'll enable `live` setting in the stage.
On interactive mode, it'll always ask user if they want to
use live (unless `--with=live` is specified), and then it'll
skip metrics/plots question and moves on to ask for path to
dvclive path.
Also, if live setting is enabled in the stage, it'll set the
`models` to be a checkpoint output.
* s/with/type and don't ask for live interactively
1. update ls __init__.py to output pretty JSON format
2. introduce write_json function into ui __init__.py
Co-authored-by: Zhenyu Gao <[email protected]>
This was used before on opening http urls as a stream,
but we no longer use those (as we use fsspec's open method).
One thing to note however is that, compared to previous
implementation, fsspec's http filesystem's open is not very
robust to failures.
speedcopy only patches shutil.copyfile, which we don't even use
anywhere because it lacks callbacks/progress and we instead use
our own copyfile implementation, so this is totally useless for
us and can be removed.
```
2021-09-21T00:15:31.6789059Z ____________________ ERROR at setup of test_get_url[azure] _____________________
2021-09-21T00:15:31.6790443Z [gw0] linux -- Python 3.7.12 /opt/hostedtoolcache/Python/3.7.12/x64/bin/python
2021-09-21T00:15:31.6791202Z
2021-09-21T00:15:31.6792291Z request = <FixtureRequest for <Function test_get_url[azure]>>
2021-09-21T00:15:31.6793561Z
2021-09-21T00:15:31.6794120Z def fill(request):
2021-09-21T00:15:31.6794792Z item = request._pyfuncitem
2021-09-21T00:15:31.6795627Z fixturenames = getattr(item, "fixturenames", None)
2021-09-21T00:15:31.6797090Z if fixturenames is None:
2021-09-21T00:15:31.6797922Z fixturenames = request.fixturenames
2021-09-21T00:15:31.6798497Z
2021-09-21T00:15:31.6799393Z if hasattr(item, 'callspec'):
2021-09-21T00:15:31.6800256Z for param, val in sorted_by_dependency(item.callspec.params, fixturenames):
2021-09-21T00:15:31.6801309Z if val is not None and is_lazy_fixture(val):
2021-09-21T00:15:31.6802233Z > item.callspec.params[param] = request.getfixturevalue(val.name)
2021-09-21T00:15:31.6802912Z
2021-09-21T00:15:31.6803937Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pytest_lazyfixture.py:35:
2021-09-21T00:15:31.6804952Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-09-21T00:15:31.6806085Z /home/runner/work/dvc/dvc/tests/remotes/azure.py:102: in azure
2021-09-21T00:15:31.6807090Z container.get_container_properties()
2021-09-21T00:15:31.6808748Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/core/tracing/decorator.py:83: in wrapper_use_tracer
2021-09-21T00:15:31.6822551Z return func(*args, **kwargs)
2021-09-21T00:15:31.6824143Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/storage/blob/_container_client.py:497: in get_container_properties
2021-09-21T00:15:31.6825038Z process_storage_error(error)
2021-09-21T00:15:31.6825652Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-09-21T00:15:31.6825908Z
2021-09-21T00:15:31.6826916Z storage_error = HttpResponseError("Operation returned an invalid status 'The value for one of the HTTP headers is not in the correct format.'")
2021-09-21T00:15:31.6827586Z
2021-09-21T00:15:31.6828351Z def process_storage_error(storage_error): # pylint:disable=too-many-statements
2021-09-21T00:15:31.6829086Z raise_error = HttpResponseError
2021-09-21T00:15:31.6829597Z serialized = False
2021-09-21T00:15:31.6830066Z if not storage_error.response:
2021-09-21T00:15:31.6830554Z raise storage_error
2021-09-21T00:15:31.6831558Z # If it is one of those three then it has been serialized prior by the generated layer.
2021-09-21T00:15:31.6832480Z if isinstance(storage_error, (PartialBatchErrorException,
2021-09-21T00:15:31.6833636Z ClientAuthenticationError, ResourceNotFoundError, ResourceExistsError)):
2021-09-21T00:15:31.6834687Z serialized = True
2021-09-21T00:15:31.6835753Z error_code = storage_error.response.headers.get('x-ms-error-code')
2021-09-21T00:15:31.6836524Z error_message = storage_error.message
2021-09-21T00:15:31.6837001Z additional_data = {}
2021-09-21T00:15:31.6837398Z error_dict = {}
2021-09-21T00:15:31.6837739Z try:
2021-09-21T00:15:31.6838506Z error_body = ContentDecodePolicy.deserialize_from_http_generics(storage_error.response)
2021-09-21T00:15:31.6839345Z # If it is an XML response
2021-09-21T00:15:31.6839837Z if isinstance(error_body, Element):
2021-09-21T00:15:31.6840285Z error_dict = {
2021-09-21T00:15:31.6840750Z child.tag.lower(): child.text
2021-09-21T00:15:31.6841435Z for child in error_body
2021-09-21T00:15:31.6841822Z }
2021-09-21T00:15:31.6842195Z # If it is a JSON response
2021-09-21T00:15:31.6842692Z elif isinstance(error_body, dict):
2021-09-21T00:15:31.6843413Z error_dict = error_body.get('error', {})
2021-09-21T00:15:31.6843893Z elif not error_code:
2021-09-21T00:15:31.6844347Z _LOGGER.warning(
2021-09-21T00:15:31.6845611Z 'Unexpected return type % from ContentDecodePolicy.deserialize_from_http_generics.', type(error_body))
2021-09-21T00:15:31.6846887Z error_dict = {'message': str(error_body)}
2021-09-21T00:15:31.6847294Z
2021-09-21T00:15:31.6847730Z # If we extracted from a Json or XML response
2021-09-21T00:15:31.6848205Z if error_dict:
2021-09-21T00:15:31.6848792Z error_code = error_dict.get('code')
2021-09-21T00:15:31.6849704Z error_message = error_dict.get('message')
2021-09-21T00:15:31.6850578Z additional_data = {k: v for k, v in error_dict.items() if k not in {'code', 'message'}}
2021-09-21T00:15:31.6851392Z except DecodeError:
2021-09-21T00:15:31.6851801Z pass
2021-09-21T00:15:31.6852290Z
2021-09-21T00:15:31.6852601Z try:
2021-09-21T00:15:31.6853156Z # This check would be unnecessary if we have already serialized the error
2021-09-21T00:15:31.6853821Z if error_code and not serialized:
2021-09-21T00:15:31.6854423Z error_code = StorageErrorCode(error_code)
2021-09-21T00:15:31.6855164Z if error_code in [StorageErrorCode.condition_not_met,
2021-09-21T00:15:31.6856229Z StorageErrorCode.blob_overwritten]:
2021-09-21T00:15:31.6856938Z raise_error = ResourceModifiedError
2021-09-21T00:15:31.6857739Z if error_code in [StorageErrorCode.invalid_authentication_info,
2021-09-21T00:15:31.6858641Z StorageErrorCode.authentication_failed]:
2021-09-21T00:15:31.6859468Z raise_error = ClientAuthenticationError
2021-09-21T00:15:31.6860256Z if error_code in [StorageErrorCode.resource_not_found,
2021-09-21T00:15:31.6861256Z StorageErrorCode.cannot_verify_copy_source,
2021-09-21T00:15:31.6861998Z StorageErrorCode.blob_not_found,
2021-09-21T00:15:31.6862716Z StorageErrorCode.queue_not_found,
2021-09-21T00:15:31.6863718Z StorageErrorCode.container_not_found,
2021-09-21T00:15:31.6864526Z StorageErrorCode.parent_not_found,
2021-09-21T00:15:31.6865432Z StorageErrorCode.share_not_found]:
2021-09-21T00:15:31.6866549Z raise_error = ResourceNotFoundError
2021-09-21T00:15:31.6867609Z if error_code in [StorageErrorCode.account_already_exists,
2021-09-21T00:15:31.6868499Z StorageErrorCode.account_being_created,
2021-09-21T00:15:31.6869383Z StorageErrorCode.resource_already_exists,
2021-09-21T00:15:31.6870728Z StorageErrorCode.resource_type_mismatch,
2021-09-21T00:15:31.6871688Z StorageErrorCode.blob_already_exists,
2021-09-21T00:15:31.6872429Z StorageErrorCode.queue_already_exists,
2021-09-21T00:15:31.6873252Z StorageErrorCode.container_already_exists,
2021-09-21T00:15:31.6874124Z StorageErrorCode.container_being_deleted,
2021-09-21T00:15:31.6874922Z StorageErrorCode.queue_being_deleted,
2021-09-21T00:15:31.6875857Z StorageErrorCode.share_already_exists,
2021-09-21T00:15:31.6876608Z StorageErrorCode.share_being_deleted]:
2021-09-21T00:15:31.6877289Z raise_error = ResourceExistsError
2021-09-21T00:15:31.6877807Z except ValueError:
2021-09-21T00:15:31.6878259Z # Got an unknown error code
2021-09-21T00:15:31.6878648Z pass
2021-09-21T00:15:31.6878942Z
2021-09-21T00:15:31.6879424Z # Error message should include all the error properties
2021-09-21T00:15:31.6879908Z try:
2021-09-21T00:15:31.6880416Z error_message += "\nErrorCode:{}".format(error_code.value)
2021-09-21T00:15:31.6881289Z except AttributeError:
2021-09-21T00:15:31.6883082Z error_message += "\nErrorCode:{}".format(error_code)
2021-09-21T00:15:31.6883807Z for name, info in additional_data.items():
2021-09-21T00:15:31.6884446Z error_message += "\n{}:{}".format(name, info)
2021-09-21T00:15:31.6884932Z
2021-09-21T00:15:31.6885741Z # No need to create an instance if it has already been serialized by the generated layer
2021-09-21T00:15:31.6886538Z if serialized:
2021-09-21T00:15:31.6887019Z storage_error.message = error_message
2021-09-21T00:15:31.6887526Z error = storage_error
2021-09-21T00:15:31.6887908Z else:
2021-09-21T00:15:31.6888499Z error = raise_error(message=error_message, response=storage_error.response)
2021-09-21T00:15:31.6889380Z # Ensure these properties are stored in the error instance as well (not just the error message)
2021-09-21T00:15:31.6890078Z error.error_code = error_code
2021-09-21T00:15:31.6890630Z error.additional_info = additional_data
2021-09-21T00:15:31.6891956Z # error.args is what's surfaced on the traceback - show error message in all cases
2021-09-21T00:15:31.6892672Z error.args = (error.message,)
2021-09-21T00:15:31.6893112Z try:
2021-09-21T00:15:31.6893784Z # `from None` prevents us from double printing the exception (suppresses generated layer error context)
2021-09-21T00:15:31.6894873Z > exec("raise error from None") # pylint: disable=exec-used # nosec
2021-09-21T00:15:31.6896126Z E azure.core.exceptions.HttpResponseError: The value for one of the HTTP headers is not in the correct format.
2021-09-21T00:15:31.6897385Z E RequestId:d83b8102-bde1-4a77-aa7a-b67bbc2c4a80
2021-09-21T00:15:31.6898080Z E Time:2021-09-21T00:09:53.815Z
2021-09-21T00:15:31.6898578Z E ErrorCode:InvalidHeaderValue
2021-09-21T00:15:31.6899259Z E headername:x-ms-version
2021-09-21T00:15:31.6900066Z E headervalue:2020-10-02
2021-09-21T00:15:31.6900355Z
2021-09-21T00:15:31.6901781Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/storage/blob/_shared/response_handlers.py:177: HttpResponseError
```
* rebased PR 6111 onto master
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use make_tmp_dir fixture in tests
Co-authored-by: U119511 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Alexander Meier <[email protected]>
Co-authored-by: meierale <[email protected]>
* tests: additional callbacks test for remaining filesystems
* requirements: bump pydrive2 to 1.9.4
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Trying to see if getting rid of this dependency, our issue with timeout
gets solved or not.
We use timeout-minutes anyway on GHA, so we probably don't need it.
Locally, users can monitor this themselves.
* machine: add `machine list`&`machine modify`
1. add new command `dvc machine list`
2. complete `dvc machine modify`
3. add new unit tests for this two commands to ensure the call.
4. add functionality tests for two commands
* Update dvc/config_schema.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update tests/func/machine/test_machine_config.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some reviewed problems
1. capitalize global variables.
2. remove precision restriction.
3. solve windows path seperator problem.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* dvc cli: fixed error in 'dvc config --list' in non repo dir
Fixes#6496
* Issue 6496: Added test that covers the issue
* dvc cli: fixes error in 'dvc config <name>' in non dvc repo dir
Fixes the specified error. Did some refactoring based on previous commits.
Also added a test case for setting config in non dvc repo dir.
Fixes#6496
* Issue-6496: dvc config --<level> and dvc config --list --<level> will now throw config error if ran in non dvc repo with repo only level
* Issue 6496: Segregated config levels into repo and system levels that helped in simplifying the logic to decide appropriate level at the time of cli operation.
The current ci failure was caused by wrong order of the experiments.
Because our test script runs so fast that they might share the same
commit timestamp, that makes them in an uncertain order. And then gives
a wrong branch_base.
* output: use remote kwarg instead of odb
Fixes#6596
* Update dvc/output.py
Co-authored-by: Dave Berenbaum <[email protected]>
* add test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Dave Berenbaum <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fs.download: add callback support
* fix lint issues
* compat for gdrive, use fs to upload text for tests
* fix typing issues
* skip gdrive test
* Update tests/func/test_fs.py
* fs.upload: add support for optional callback
Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Show csv format of experiments
fix#5989 and #5446
1. add --show-csv to dvc exp show
2. add a new functional test for csv format
3. add a new unit test for show_experiements
4. solve name duplication in table common headers (Created,Parent, etc... ), metrics files and param files.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Set timeout for jobs in Github Actions
Sometimes tests are getting frozen(most likely an issue with `pytest-xdist`), and sometimes the pip resolver is slow due to some weird constraints set by us or one of our dependencies.
The default timeout is 360 minutes (6hrs), but our test completes on ~18 minutes at maximum, so I think 30mins should be enough to account for occasional CI slowdown.
Also keeping the total job timeout at 45 mins, we usually require just 4 mins of extra time to do installation and setup but giving it 11 more minutes just in case.
Similarly, `lint` completes in <5 mins usually, but giving it a 10 mins timeout.
This should give us early feedback on the issue.
* Update setup.cfg
* Update setup.cfg
* machine: install dvc via startup_script by default
* machine: support configuring instance type and startup script
* accept startup_script as a config relpath
* use pushd/popd in default install script
* extract transfer related logic from odb to fs.utils.transfer
* don't makedirs on dirinfo
As local odb needs to set a mode for dir, we cannot create a directory
in the transfer
Making sure that script takes a bit more time than the await time
I am not really sure this was the cause though.
Also the tests are flaky only in MacOS, but I haven't been able to reproduce.
* Clean up remotes's exps
fix#6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail
Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
* Added dvclive to tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Moved dvclive to rest_requirements
* Update setup.py
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Added refresh tag
* Formatted
* Added refresh_seconds to live
* Added refresh test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add refresh_seconds to render
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update html.py
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Update README.rst
Fixes typos and makes the related technologies stand out from their description.
* Apply suggestions from code review
* Update README.rst
* Wrapped modified portions to 80 chars or less
Left others untouched to simplify review
Co-authored-by: Jorge Orpinel <[email protected]>
* added auth to iter_url and created custom open for http fs
* added default value to auth
* Update dvc/utils/http.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/fs/http.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* pr fixes
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Remove all of the experiments in the workspace
fix#5676
1. remove all the experiments in workspace
2. add tests for it
* Only remove baseline ones
* Add a new arguments --all
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Rename and shortcut for these commands
* Solve the issue from removing a changing list.
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* delete workspace option
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Replacing DVC_ANALYTICS. This is following our other env conventions and
will not have effect if set to false/no/etc, and will only work if set
to true/yes/y/etc. Will disable analytics even if it is explicitly
enabled in our configs.
Kudos @skshetry
We've used updater for years now to notify our users ASAP about new
releases. These days dvc is more stable and we've actually been
receiving numerous complains about updater being too annoying and
even introduced a config flag to turn it off, which a lot of people do
and which defeats its purpose. Instead of introducing some complex logic
that would notify more rarely and once for the user instead of once for
every dvc project they have - this PR just turns it off for everything,
except `dvc doctor/version`, which is the command that people use when
they are running into issues and that will give them a hint to just try
out the newest version first, as it might be already fixed.
This PR is a more of a compromise, to get the most out of updater while
removing a pain point of it being too annoying, without investing much
time into it and not having to maintain complex unnecessary logic. We
might get back to introducing something more intricate in the future,
if the need arises.
Fixes#5912
* add support for Python 3.10
pyarrow is not available, so we are skipping tests for it.
And, restricting `hdfs` extras to be not installed in Python3.10.
Pathlib removed `_init` method and `init` parameter, so just working
around that. It's not a big change.
* skip windows on 3.10; update pip/wheel/setuptools during install
pywin32 is not yet available for 3.10 Python, so skipping those for now
* install libgit2 with Python3.10 as pygit2 wheel is not available
* install libgit2
* only run on macOS, libgit2-1.1 not available on ubuntu
* Remove a special queued experiments
fix#6157
1. dvc exp remove not accept queued experiments name
2. add some tests for this feature
* Extract tests and add revision support
1. Extract remove experiments to a new file.
2. revision can be used to remove special queued experiment
* Accept shortened revisions
* Split removing committed and queued exp functions
* Api name change
* Return to the old API name
* shorten some functions
* Error message change
* Better test cases, more corner case
* Still raise exception in a mixed case
* Push running checkpoint to remote
Fix#6182
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Get the branch name
* Finish this PR
1. move env name to dvc/env.py.
2. add some tests for it.
* Update after finally commit
* Some problems found in the review
1. change the behavior of self remote
2. do not use string DVC_EXP_AUTO_PUSH
3. downgrade the logger level in auto push
4. use full branch ref
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* add a remote check before auto push
* Split current one env into two
* Rename DVC_EXP_CHECKPOINT_PUSH to DVC_EXP_AUTO_PUSH
* value error
* Remove getenv_bool switch to env2bool
* Better on handling Dulwich exceptions
* Name changing removing prints
* Update dvc/env.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some changes in review
* Add validation to repo URL
* Add new type of SCMError
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Use spy tests call counts and args
* Update dvc/scm/git/backend/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/scm/git/backend/dulwich.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove validate_git_repo to validate_git_remote
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* demonstrate issue while sending analytics report
When sending analytics, we are not flushing the data to the temp file
which causes failure at times, especially on Windows
* close file before calling daemonized analytics send command
This way, the contents bufferred will get a chance to be flushed.
* Remove rwlock decorator for the run() method of the Stage class.
Instead of pre-emptively grabbing read locks on the dependencies
and write locks on the outputs when calling the run() method,
delegate the locking mechanism to other methods.
This results in more selective locking that is less likely
to cause a LockError when executing steps in parallel.
* Remove rwlocked decorator from reproduce() method of Stage class.
* Reinstate read-lock (not write) for run method of Stage class
* fix bugs and typos
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* objects.status: use object IDs instead of naive objects
* objects.transfer: use object IDs instead of naive objects
* objects: add ReferenceHashFile
* odb: add ReferenceObjectDB
* repo: collect used object IDs instead of naive objs
* support basic/naive fs serialization
* objects.stage: stage file objects in addition to trees
* objects: remove save(), update transfer()
* output/dep: update for osave removal
* path_info: support both url and path attributes
* status/gc: use object IDs instead of naive objs
* diff: update staging usage
* commit: update staging usage
* tests: clean staging when needed
* tests: update for save/transfer/stage changes
* dvcfs: don't allow config serialization (repofs should be used instead)
* utils: return mtime as str, size as int
* tests: update unit tests for save/stage/transfer changes
* objects: use Repo.open when needed in ReferenceHashFile
* fix ls test staging
* objects.stage: don't skip hash check when loading from state
- individual hashes will still be loaded from state if they are still
valid
* ReferenceHashFile: serialize repo fs paths relative to repo root
* preserve state when staging
* objects.tree: skip hash computation in digest() when we have already
loaded hash from state
* always check file existence in referencehashfile
* update repofs test since stage no longer allows nonexistent files
* odb: only protect on check() if hash was verified
* staging: use unique memfs namespace for staging per dest ODB
* tests: clean full staging namespace
* refodb: preserve repofs cache dir
* repodependency: stage imports to local ODB and strict check circular
imports
* fetch: fetch staging last
* staging: make dest odb mandatory and preserve state
* refodb: cache nonlocal fs's
* fix rebase errors
* repofs: support full instantiation from fs config
* repofs: preserve subrepo factory for erepos
* tests: clarify repofs dirty file behavior
* Fix merging of params when empty
Recent release of python-benedict==0.24.1 started ignoring empty dictionaries
during merge. The way we expected to work was questionable and also how
benedict itself tries to handle the references to empty dictionary is
also questionable.
I consider this to be a proper way to handle this even if this gets fixed
on the benedict side.
* Revert "setup: limit python-benedict version (#6377)"
This reverts commit 15de826266.
* add comment for why passing through benedict
add additional excludes to DefaultAzureCredential
which are needed in a Windows Environment to
use az cli provided credentials, etc.
Co-authored-by: Alexander Meier <[email protected]>
* objects: migrate remote status to objects.status
* data_cloud: use objects.status
* objects: migrate remote._process to objects.transfer
* data_cloud: use objects.transfer
* objects: handle memfs staging objects in transfer
* fetch/import: use unified transfer()
* update get_remote and transfer exception usage
* update tests
* objects.transfer: use src ODB verify rules after xfer
* odb: move index from remote into odb
* objects.transfer: skip src status check when possible
* update tests
* remove dvc.remote
* odb.index: migrate to diskcache
* objects: fix status/transfer optimization
* update index usage
* state: remove dead sqlite related code
* update cloud tests
* drop unnecessary remove in tests
* use abstract base class in odb index
* status: load trees from local cache instead of remote odb when possible
* handle status when dir cache is explicitly removed
* params/metrics/plots/exp show: introduce onerror
* fixup
* plots/metrics/params: removing errors
* removing more errors
* removing errors
* some more removing
* new plots ux
* new ux for metrics
* new ux for params
* fixing tests
* fixups
* minor test fixes
* minor test fixes
* commands: plots/params: fix ui imports
* working on error handlng decorator, stopped at show checkpoints
* error_handler, further dvelopment
* plots just before moving loading to collect
* appending errors to results: initial passing
* fixup
* onerror: remove onerror fixture
* fixups
* plots/metrics/params/experiments: temporarily remove error warnings
* plots: extract parsing out of class method
* metrics/experiments: provide exception serializer for show-json
* params/plots/metrics: signal errors
* rebase fixups
* fix TODOs
* fixup
* skshetry's review refactor
* Migrate to ossfs
fix 6177
* Add verification to object stored on oss
* Change the oss remote's dependency.
pass all of the tests
* Performance optimaztion and ossfs version update
* Remove the argument to set the resumable get/put
1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default
* Use a new api for performance optimization
* Without filesize progress bar version
* With progress bar version
* Bump and add comments
1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger
* Change the name of kwrags following parent class.
* Bump to new ossfs and rewrite upload_fobj
Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj
* Optimization on rm_files
* Some optimization follow the old version.
* Update dvc/fs/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/objects/db/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* bump into ossfs 2021.7.2
* Bump into ossfs 2021.7.3OC
* Switch to real_oss_server
* Switch to a real oss server
* Use the oss bucket located in US.
* Remove some emulator related code
* Add envs to ci
* remove some oss pytest XFAIL
Because we didn't use emulator, we can remove some pytest XFAILs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* exp: allow fetching from arbitrary executor by git URL
* exp: write info for running executors to .dvc/tmp/exp/run
currently only includes PID and git fetch URL
* exp: refactor stash/queue cleanup
* exp: add ability to collect dict of currently running experiments
* exp show: display run/queue state in the table
* exp run: unlock main repo while running tmpdir executors
* add tests for new exp show behavior
* fix workspace checkpoint run handling
* fix issue where running stash revs were shown as queued
* use "running"/"queued"
* add executor (location) column
* compare: add TabularData.is_empty
* use td.is_empty to drop state/executor
* use "workspace" and "temp" as executor names
* fs: use fsspecwrapper in memfs
* odb: add memfs staging ODB in ODBManager
* odb: add read_only attribute for odb instances
* objects: handle memfs paths when staging trees
* odb: remove git ODB
* output: use staging ODB
* RepoDependency: use staging odb for imports
* dvcfs: handle case where dir cache is unavailable
* fix circular import check
* odb: use temp local ODB for staging
* objects: use staging ODB in stage() and tree load
* objects: add typing for add/check/load/stage
* output/dep: use new staging
* add unit tests for local odb staging
* localfs: account for string paths in move()
* include local in odb.by_scheme
* tests: count dir cache like any other object in test_pull_git_imports
* odb: clear staging on gc()
* objects: add tree.get/filter/insert
* output: use obj.filter/get for granular dir commit
* squash filter logic output
* dependency: don't filter imports at the dep level
* don't init staging ODB for external cache
* use staging ODB when computing tree hash for external outs
* remove unneeded force in test granular commit
* use unique-per-manager memfs URLs for staging
* odb: make staging per-ODBManager
* objects: move state dependent staging into ODBManager
* objects: support checking multiple odbs
- staging dependent checks can be done via ODBManager.check
* checkout: do state lookup before explicit stage() in _changed()
* output: use odbmanager based staging
* RepoDependency: use odbmanager based staging
* update obj staging unit tests
* checkout: remove check(), handle individual file exceptions on link()
* diff: use odbmanager based staging
* objects: add ObjectPermissionError for read/write perm errors
* odb: move staging out of odbmanager
* objects: handle staging ODB in objects.stage()
* objects.tree: don't stage anything in tree methods
* odb: clear staged objects on gc()
* remote: don't filter objs by scheme, objs are already grouped by ODB
* diff: update objects.stage usage
* output/dep: update stage() usage
* update ODB tests
* update func tests
* remove memfs from FS_MAP
* revert unneeded objects.check() changes
* remove unused tree.insert()
* revert unused odbmanager changes
* add separate odb error classes
* stage external outs directly in ODB
* check for obj existence in main ODB before staging
* use .dvc/tmp/staging instead of per-odb staging
* gc: only gc loal staging once
Kudos @skshetry for discovering that we've been publishing py2.py3
wheels on pypi and researching the cause.
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
* Update s3.py
pass verify flag when creating a session
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fixed diff summary to only show states with values.
* Removed redundant comma by using join.
* Swithced for loop to generator for better readability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* update dvc exp run -h message
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* add: do not delete stage files before add
Because of the way we collect stages and cache them, we were not able to
collect them for the `add` without removing them from the workspace.
As doing so, we'd have two same/similar stages - one collected from the
workspace and the other just created from the `dvc add` in-memory.
This would raise errors during graph checks, so we started to delete them
and reset them (which is very recently, see #2886 and #3349).
By deleting the file before we even do any checks, we are making DVC
fragile, and results in data loss for the users with even simple mistakes.
This should make it more reliable and robust.
And, recently, we have started to keep state of a lot of things, that by
resetting them on each stage, we waste a lot of performance, especially
on gitignores. We cache the dulwich's IgnoreManager, which when resetted
too many times, will waste a lot of our time just collecting them again
next time (see #6227).
It's hard to say how much this improves, as this very much depends on
no. of gitignores in the repo (which can be assumed to be quite in number
for a dvc repo) and the amount of files that we are adding
(eg: `-R` adding a large directory).
On a directory with 10,000 files (in a datadet-registry repo),
creating stages on `dvc add -R` went from 64 files/sec to 1.1k files/sec.
* add tests
* make the test more specific
* output: str only once
Calling `__str__` takes a long time and if we are dealing with a large
dataset, it will result in a very substantial time. For example,
locally, for 700K objects, it took 20sec more.
Related to #6276
* output: don't use os.path.join
Takes around 2sec for 700K objects.
* output: use fs.sep
* output: remove unused sep
* separate validation and transformation of args from add func
* make tests pass for Python3.6
* cleanup
* use dict for attr-access and patch boundarguments.arguments
* Keep trailing slash when checking if directory is gitignored
* Calling is_ignored with correct type (could be Path-like)
* Fix test to comply to annotated types
* Fix separator issues
* Relaxed typing in is_ignored
* Added test
* Test scm
* Fix test on Windows
* Expanded test_ignored_dir_unignored_subdirs
* Even more assertions
* objects: add ExternalRepoFile object type
* RepoDependency: implement get_obj (use ExternalRepoFile)
* drop distinction between used external and used objects
* update tests
* remove unneeded fetch from ExternalRepoFile
* wrap erepo errors as an object error
* update cloud/remote functions for new used_objs behavior
* fix update() rev/locked behavior
* update import tests
* remove ExternalRepoFile
* move get_obj and get_used_objs implementations back into repo dependency
* return objects plus associated ODB (remote) in get_used_objs
* odb: add tmp_dir field and support creating Remote from an odb
* return dict mapping odb -> objects in get_used_objs
* use dummy git odb for git imports
* fetch: support pulling from specific odb
* update cloud funcs for get_used_objs usage
* update tests for used_objs behavior
* use fetch in dep.download
* catch circular imports from local fs repos
* add tests for chained and circular imports
* move tmp_dir into odb config
* update return docstring for repo.used_objs
* Add Studio to the related technologies list
* Add CML to DVC Studio description
* fix k8s names a bit
Co-authored-by: Ivan Shcheklein <[email protected]>
* add: prevent crash while getting relpath
Since python 3.8 os.path.realpath has resolved windows network shares
to UNC paths. This breaks the logic for relpath as the UNC and mapped
drive paths don't have an obvious common prefix. As it is not easy to go
back to the mapped drive path from UNC path, this is resolved by always
using the UNC paths for calculating relative paths in windows.
Fixes#5832
* relpath: Update changes following review
Two changes:
- use pytest.mark.skipif rather than if statement inside test to make
it clearer that test is windows specific
- Use tail rather than upper_path to denote rest of path after drive
name taken off (to match python documentation)
Pull request: #5837
* tests, linter: update to add test case and apply linter though pre-commit
Co-authored-by: Tom Rutter <[email protected]>
* objects: make HashInfo & HashFile hashable
* stage/output: collect objects (HashFiles) instead of NamedCache
* remote/data_cloud: use objs instead of NamedCache
* repo: use objects instead of NamedCache
* rerun black after rebase
* remove NamedCache usage from run-cache
* drop NamedCache
* update HashInfo.__hash__
* update unit tests
* revert remote_name change
* start cleaning up test failures
* fix tree obj expansion
* update missing cache logging
* fix external dict handling
* update naive odb.gc
* update state save behavior for pull
* update test_data_cloud
* update test_remote
* fix gc with multiple repos
* objects.tree: support filtering w/preserved hash_info
* fix stage cache restore
* drop fs.path_info from HashFile.__hash__
* objects: add optional name attribute
* revert missing cache logging name/suffix changes
* revert name changes in remote tests
* review fixes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update dvc/remote/base.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* config, remote: Made S3 CA bundle customizable
botocore allows a path to a custom CA bundle either by passing a path
to the CA bundle file into the verify argument of
boto3.session.Session.client or passing None (the default) which will
fall back to the AWS config. Previously, the DVC config only accepted a
boolean into the ssl_verify option in the remote S3 config. This changes
the DVC config to accept both string and None in addition to boolean and
defaults to None. I also changed the default for ssl_verfiy to None in
BaseS3FileSystem. Thus, if ssl_verify is not provided, botocore will
fall back to the AWS config.
Testing
Unit tests to cover the changes to the config schema and addition
ssl_verify types that will be passed into S3FileSystem. Also, ran
dvc push -r object-store data/cifar-10-python.tar.gz
in my work environment that has a private S3 endpoint that requires a
custom CA bundle, both with and without ssl_verify specified in the
config. This was successful, showing that communication could be
established. And I ran
dvc remote modify object-store ssl_verify "$HOME/.aws/cabundle.pem"
and confirmed that the custom CA bundle path was added to the config.
Fixes#6012
* Removed default None on ssl_verify
Responding to PR comment, removed the Optional, default None on
ssl_verify since the config keys are optional by default. Rather
than a missing ssl_verify producing a None that eventually gets
filtered, it doesn't appear in the parsed config in the first place.
Co-authored-by: Robert Van Wesep <[email protected]>
astroid==2.5.7 has improved inference and was throwing
errors that were false negatives before. Fixing these
and ignoring some.
Also added `funcy.decorator` to the `signature-mutators`,
as it changes function signature (i.e. provides `Call` as
a first argument which is not required to pass).
Example errors that we were getting because of this:
```
tests/remotes/gdrive.py:40:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
************* Module dvc.stage.serialize
dvc/stage/serialize.py:38:1: E1120: No value for argument 'func' in
function call (no-value-for-parameter)
************* Module dvc.fs.http
dvc/fs/http.py:19:1: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
************* Module dvc.fs.gdrive
dvc/fs/gdrive.py:66:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
dvc/fs/gdrive.py:459:5: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
```
Pre-requisite for removing fs.path_info and switching to regular strings
as paths in filesystems.
With new filesystems we no longer need the zoo of different dependency
and output types, as most of them are now just handled by the underlying
filesystem. Though deps/outs are still due for a refactor in other
aspects.
* Decoupling dvcignore and fs
[x] remove dvcignore from fs objects
[x] attach dvcignore to repo
[x] add ignore check to functions(add, checkout, output, pipeline, etc)
[x] pass all tests
[x] final check
* Solve all of the arugments error.
* add dvcignore to dvc list
* Solve some test_ignore cases
* add dvcignore to checkout.
1. add dvcignore to 'checkout', 'add'
2. add ignore_filter to fs
* Solve two tests
* Get rid of ignore_filter
* Add ignore examine to download
* Solve a test issue
* Solve dvcignore and subrepo problems
* Solve sub repo error
* Revert some changes
* pass all tests
* Remove debug codes
* Some final exams
* dvcignore trie tree only update when needed
* reuse of the dir result in path walk.
* Some change after code review.
* Recover something from false `--force` push
* add abspath trans to dvcginore
* pass pylint
* pylint related
* Update dvc/fs/repo.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Changes related to reviews.
* dvcignore call api change
from dvcignore.walk_files(fs.walk(path_info)) to dvcignore.walk_files(fs, path_info)
* Get rid of PathInfo in test_ignore.py
* fs.walk follow the arguments of os.walk
* Dvcignore as parameters
* pass dvcignore as arguments to checkouts
* Decoupling state and repo
* Update dvc/dvcfile.py
Co-authored-by: Paweł Redzyński <[email protected]>
* Update dvc/objects/stage.py
Co-authored-by: Paweł Redzyński <[email protected]>
* Solve some small problems
* Add tests to verify that dependency now not ignore
fix#5256
* Update dvc/ignore.py
* Update dvc/fs/git.py
* Update dvc/repo/__init__.py
* Some small improvements
* pass black
* Refactor is_ignored
* Update dvc/output/base.py
* repofs: use dvcignore in exists/isdir/isfile
Per https://github.com/iterative/dvc/pull/5812/files#r630604524
Co-authored-by: myname <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
We used to allow changing default output and error stream
but that was problematic for the testing. Now it just uses
sys, so capsys should work for the testing.
The proper way to change sys.stdout and sys.stderr is
by using 'redirect_stdout' now.
* try to close dvc.state on exit
I have a feeling that it is causing the issues with
shutil.copytree and some PermissionError during 'init',
as we are not closing it between multiple 'main()' calls.
But I am not quite sure, and this seems like a good thing to
do anyway.
* fixup
* close dvc on exit
* close cache on tests and on init
* azure: use a custom event loop for authentication
* add test_temp_event_loop
* wrap open() too!
* ignore aiohttp warnings for normal level use-cases
* rename tests/unit/remote/test_azure to tests/unit/fs/test_azure
we just need to mark them with needs_internet marker.
Unfortunately, it's not possible to do that automatically
with the fixtures as fixtures are per-session and we need
something on more higher level (which the marker is not passed
through in the pytest).
This is because, the repo root is used from a realpath, and the stage.wdir is abspath.
We might need to use same kinds of path, but this patch seems to be the easiest/fastest
way to fix the flaky tests (if all the tests agree).
* dvcfile: add check for gitignored when validating filenames
* update tests
* update exception using in tests
* review fixes
- clarify method name
- use FileIsGitIgnored
I tried increasing max_run to very high number: 1000. Even in those
cases, it failed 947 times. So I don't really see a reason to mark
it as flaky. Anyway, we are treating it as xfail by keeping it failing
in the CI, so this PR does not change anything for us. :)
I found that it's harder to style a header with the list using index,
they are to be applied by header name. Otherwise, we might end up using
a different header for coloring, or worse throw some error because the
header name and style don't match.
Row does have it's property of index, so it makes sense there.
* apply: allow applying exp with multiple possible branches
* default to new checkpoint branch after applying ambiguous commit
* output shas in addition to exp names by default
* update show tests
* git: allow force checkout in detach_head context
* exp run: error when trying to resume from a non-head checkpoint without
modifications
* add test for apply/resume from intermediate checkpoint
* list some possible heads when multiple matches
* use new table data structure for diff and show
* fix params diff showing changes
* metrics show: use default precision
* skip precision format for exp diff params
* exp diff: separate tables in --quiet mode
* change precision to signify significants digits
changed f specifier to g, so that we show significant digits
instead of fractional digits. This does changes how precision
worked for metrics/params diff/show.
* stage,commit: add support for only committing data stages from
experiments
* exp run: `dvc commit` data deps when stashing exp changes
* add test for modified DVC-tracked data when generating experiment
* test with and without params change
* handle run without target
* objects: merge transfer() with stage()
* objects: drop streamfile
* objects: preserve the signature of get_file_hash
* objects: override hash_jobs if jobs= passed
* objects: support no progress bar directive
* tests: spy on _get_file_hash
* objects: update=... directive
* update: only transfer missing caches with --to-remote
* tests: remove local_remote from the primary test (due to it moves)
* objects: add move=True/False option
* objects: implement _get_file_obj on top of get_file_hash
* objects: use threading on save()
* objects: pass jobs= on to-cache as well
* objects: use in place tqdm
* objects: check for FileExistsError
* Update dvc/objects/__init__.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Due to https://about.codecov.io/security-update/ , there is a potential that we've leaked these. This PR takes a more granular approach when setting secret env vars so that this doesn't happen in the future.
We'll change these secrets after this PR is merged.
* exp: add util method for getting baseline HEAD during run
* exp: use baseline for dvc commands which compare to HEAD during exp run
* add test for new util method
* update plots diff test mock
* Add support for kerberos token based auth
* Added kerb_ticket as attribute to HDFSFileSystem
* Lint fix for config schema
* Updated black style changes
* Removed staticmethod decorator from hdfs
This part of logic used to be only used by external repo for cloned
repos, but since we now handle local repos without cloning, we don't
always pass scm instance. This PR automatically creates scm instance
for such cases.
Fixes#5590
* params: read from ParamsDependency if possible
* params: introduce include_untracked flag
* params: include untracked upon targets
* experiments: add option to show tracked params
* fixing test
* params: rename tracked to deps
* params: roll back to default params.yaml behavior
* experiments: add option to show only used params on diff
* Update dvc/command/experiments.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/experiments.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/params.py
Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
* exp show: Allow for wildcard patterns in include and exclude params/metrics
Fixes#5642
* Add test case for nested metrics/params
* match full string in _filter_name
* pre-commit
* Update dvc/command/experiments.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Added SSL verification option
* Changed param name to avoid conflicts
* verify_ssl now accepts strings
* Added simple unit tests
* Changed parameter name fron verify_ssl to ssl_verify so it's consistent with HTTP remotes
Forget to change 1 test
* add processing on ssl_verify param
typo
* revert back to bool param
* Update dvc/config_schema.py
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
This is achieved by using `lock_repo`
context manager that can be used in a recursive depth
that only resets stages when it has left any lock_repo
blocks.
Also added some ui related tests for the status which I found
missing
Before, we used to set it during initialization. But this
made it hard to test using capsys as capsys changes the
reference to sys.stdout and sys.stderr with the mocked one
during the test. But since we are using dvc.ui globally and
setting it to a real sys.stdout and sys.stderr before capsys
comes into effect, testing was not possible with it.
Now, by using property hack, we will be using the values of
sys.stdout and sys.stderr during the time of write, so the mocking
would work and would be easier to test.
* dvc: fix breakage with rich==10.0.0
Change in the arguments of __rich_measure__.
Fixes#5722
* dvc: Fix linting breakage from rich==10.0.0
Fixes: #5733
* dvc: set lower bound for rich dependency to 10.0.0
Fixes#5722
lxml wheels are built with an old SDK, which breaks osxpkg notarization:
```
* error for path "dvc-2.0.7.pkg/dvc-2.0.7.pkg Contents/Payload/usr/local/lib/dvc/lxml/etree.cpython-37m-darwin.so": The binary uses an SDK older than the 10.9 SDK.
```
This commit fixes 3 things:
1. The path printed on the terminal is properly is quoted. This was
problematic when the output is piped to `xdg-open` if the path contained
spaces or quotes or special characters.
2. The path is resolved. This is not an issue at all, but makes the
output nicer.
3. The path with which the browser is opened with uses relative path.
This is done to solve issue in WSL, that the absolute path is not
accessible outside the WSL for host programs as the linux filesystem
is available on `/wsl$` path.
* diff: clear calculated renamed files from inverted hashmap
We were trying to reuse the first encounter of file from inverted hashmap again and again, which was throwing ValueError
* refactor
* remove ordering
* add tests
* build: add pyinstaller hook for googleapiclient
The Google API Discovery Service JSON files from
googleapiclient/discovery_cache/documents were being
excluded due to a bug and, particularly, drive.v2.json
is essential for our gdrive backend. Now we're explicitly
asking PyInstaller to embed these files on the generated packages.
Fixes#5618
* Suppress linter error for unrecognized imports
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Apply black style after manual code review
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* import-url: support importing empty directories on object storage services
Empty directories are basically files with a trailing slash at the end of their names,
with empty contents on object service providers (such as azure, google storage). This
patch implements the `is_empty()` method for the `FSSpecWrapper` to determining whether
a path is empty (directory or a file) and also tweaks `.isdir()` in order to see empty
directories.
The base class is only used by azure and gcsfs for now, but if we go on the route of
migrating normal providers (like proper filesystems, not just object storages), we might
need to do these tweaks on a subclass (like `ObjectStorageFSSpecWrapper` etc), but I
don't think it is neccesary as of now.
The `is_empty` method is already there, and also used in the importing part to
warn users that they are importing something empty (https://github.com/iterative/dvc/blob/e385e59a098cd661ea1ea4257c9a5735c5c32bc6/dvc/output/base.py#L252-L254)
so this is not a new method. Just for the reference, as far as I can locate there
is no variant of this method in the fsspec.
* tests: revert custom mkdir for cloud storages
* tests: allign assingments
* add some comments
* exp run: gracefully handle duplicate workspace runs
* exp run: fix issue where errors could be logged twice
* move ref conflict check into its own function
* dvc: get rid of DirInfo
DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.
As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.
* objects: move Tree into tree.py
The `GitPythonBackend.checkout()` method already implicitly supported
this by forwarding `**kwargs` to the backend. `pygit2` didn't support
this yet.
This patch fixes this inconsistency.
* pygit2: disable set_ref with reflog message
* git: implement is_dirty in dulwich
* pygit2: implement _stash_apply
* pygit: update to 1.5.0
* enable set_ref(message=...) for pygit2
* git: replace backend reflog_delete in favor of _stash_drop
* pygit: support stash_push with custom refs
* update tests for backend stash_drop behavior
* pygit: raise MergeConflictError on failed stash apply
* pygit: reload index from disk before stash apply
* update stash_apply checkout strategy
* mark flaky test
* gdrive: sync with pydrive's save backend changes
* bump pydrive version
* better temporary credentials managements for service accounts
* proper temp var management
* repro: skip DVC deps in SCM track stage
(DVC deps will be tracked when their own stage is repro'd)
* scm: use git status when reminding to git-track files
* sort names in reminder instead of random set order
* update unit test
* live integration: get rid of dvc from live
* live: remove from api, add test for html generation during run
* stage: run: convert monitors to class context managers
* run: make monitors run in single thread
* run: move monitor logic out of run
delete is a direct alias to rm(), though apparently it is kind of
broken on the latest release of gcsfs. This doesn't change any behavior
but only uses rm() directly instead of the given alias.
Switch from `md5` to `etag` as the checksum parameter. Due to `etag`
is not preserved while doing server side copies, this patch drops support
for external gs outputs.
* state: only load when needed
* state: don't use separate load/dump
* checkout: group link removal logic together
* state: use diskcache.Cache
* staging: use pbar and save to state
* state: don't pass the whole repo to __init__
* staging: explicitly pass state object
* fs: get rid of odb property
* state: don't close the connection to cache
* repo: close: close state
* stage:list: update output and help text
* Replaced 'Produces' with 'Outputs'
* Replaced 'Stage depends on' with just 'Depends on'
* Changes help text of --names-only and --fail
* Update dvc/command/stage.py
Co-authored-by: Dave Berenbaum <[email protected]>
* s/Generates/Reports
Co-authored-by: Dave Berenbaum <[email protected]>
* git: return backend-agnostic object for resolve_commit
* use int timestamps instead of datetime
* make branch_revs Git method instead of backend
* pygit: implement resolve_commit
* dulwich: implement add with --update
* add git backend test for add()
* fix list comprehension in dulwich status()
* allow adding single string path instead of list
* handle windows paths in add(update=...)
When trying to publish to pypi, we get:
```
Invalid value for requires_dist. Error: Can't have direct dependency:
"gdrivefs @ git+https://github.com/intake/gdrivefs.git ; extra ==
'tests'"
```
We used to think this way, but state is not really an fs concept
(fsspec's dircache is though), as it is a cache of hashes of objects
that we are staging, and the hash doesn't depend on fs, as it is usually
computed by us by reading from fs.
Pre-requisite for diskcache-based state.
This makes all cache objects go through the same logic, so that all of
the appropriate rules are applied to them (e.g. proper permissions).
Pre-requisite for #5337
* Make minor change that fixes things for some reason
While working on vscode-dvc, both mattseddon and I found that this change allows
us to run `dvc exp run`, and we cannot otherwise.
I don't exactly know what this change does, and I'm hoping someone more
knowledgable does- after which we can give this PR a better name.
* test relpaths in git backend funcs which take path lists
* gitpython: handle relpaths in reset and checkout_index
* use / for git relpaths on windows
Co-authored-by: Peter Rowlands <[email protected]>
Checkout is an operation that builds object from cache(aka object
database) in a particular tree(aka filesystem). Thus it shouldn't be
part of the object database itself nor of the filesystem.
Pre-requisite for #5337
* s3: faster directory hashing
* Introduce Tree.info() API
* tests for s3
* Implement support for azure
* New .ls() API
* remove _iter_hashes from other trees
* assert recursive
* yield strings instead of file_infos
* Add GSTree.ls()
* Add DETAIL_FIELDS to the GSTree
* WebDAVTree.ls()
* proper webdav root directory initalization
* disable arguments-differ
* webhdfs: don't use self.path_info
* azure: don't use self.path_info
* webdav: don't use self.path_info
* oss: don't use self.path_info
* http: don't use self.path_info
* gdrive: don't use self.path_info
Per https://github.com/iterative/dvc.org/pull/2064
Not changing the internal terminology because we stick to Repo in a lot
of places there and adjusting all of it would be wasteful. Maybe in the
future.
* exp show: add -n/--num to display the last N commits from HEAD
* exp run: fix empty commit with no changes
* add test for exp show with multiple commits
* allow -n greater than num commits in repo
* review fix
The `--glob` flag is parsed from the command line, but in the
`CmdDataPull` class in the invocation the parameter (`self.args.glob`)
is not passed to the `pull` method implementation. Hence, always the
default value (False) is used. This commit fixes this issue and sets the
`glob` parameter properly.
* don't allow loading stages without `cmd`
This was loosened when introducing parametrization.
But, `foreach`..`do` should be considered first
before stage's regular structure. And, cmd is made
required (as it should have been)
Related: #5371, #5370, #5312
* fix schema
* add: implement saving straight to cache
* Adjust CLI error cases to handle -o
* make tests pass on windows
* support explicit --jobs
* simplify the implementation a bit
* polish the tests
* tree: Implement upload_fobj protocol to azure
* tree: Implement upload_fobj protocol to google storage
* tree: Implement upload_fobj protocol to google drive
* tree: Implement upload_fobj protocol to hdfs
* tree: Implement upload_fobj protocol to oss (alibaba cloud)
* tree: Implement upload_fobj protocol to webhdfs (also some cleanup)
* tree: Implement upload_fobj protocol to http/https
* Add _generate_download_url() to the HTTP (for tests)
* tree: Implement a temporary upload_fobj protocol to webdav/webdavs
* Unify progress bars
* Lazily evaluate the file size, for http uploads
* Remove copy_fobj protocol and _transfer_file_as_whole
* show progress bar on cache transfer
* Simplify s3 upload
* Remove CHUNK_SIZE (except for some special cases)
* remove concurrency restrictions
* Update dvc/tree/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
When I opened it, I got confused for a second and had to double-check if it was really empty.
The empty file gives a false assurance. We do need to add one, which I'll add later, but for now,
we better remove it.
* dvcignore: Fix incorrect ignored output computation
* dvcignore: Fixed unignored patterns
Unignored patterns were shown in the dvc status output and caused Error.
Also made a bit of a refactoring in `matches` and `ignore`.
* dvcignore: Fix functional tests
Not sure if I follow, but tests assume that unignored files should trigger `check-ignore`. What was the intention?
* dvcignore: Fix stage.wdir being MagicMock
* Use `is_ignored` instead of `check_ignore`
1. use is_ignored to judge DVC ignore status.
2. use check_ignore to show ignore patterns.
* Solve tests fails
* Add a new tests
* Return to the old check_ignore logic.
* Pass the pylint
* Simplify ignore tests
Co-authored-by: karajan1001 <[email protected]>
This happens because KeyError quotes the error message.
We'd still like to keep StageNotFound being a KeyError
semantics, but workaround the __str__ implementation.
Mypy was having some problem with the previous `__str__`
override.
* Implement export-to-remote functionality
* Redistrubition of functionality from tree to remote
* Pass chunk size into the open_url
* Move functionality away from remote to cache
* Basic progress bars for both transfer types
* Proper job calculation
* Redesign .open() usage in order to work with different trees
* Align the signatures of other open() functions to accept **kwargs
* Implement chunked uploading to the ssh
* Use local cache to compute directory hash (md5)
* S3: multipart upload test
* Wrap local cache's hash_to_path with PathInfo
* Use .isdir() instead of a try/except statement for calculating the from_infos
* New CLI layout (add/import-url)
* Adjust CLI tests to accept new arguments
* tests: import_url/add func/unit tests
* Introduce cache.transfer()
* Invalid CLI combination tests
* Store dir_info key in the transfer_directory
* LocalTree.upload_fobj
* Tests for utils
* Lazy imports on some places
* --to-remote migration (and repo._transfer)
* Use proper temporary location, more efficient pbar calc
* Directory transfer logic rewrite
* Use shutil.copyfileobj()
* Rebase...
* Better tests
* Inline transfer operation to .add() / .transfer()
* Open in the 'rb' mode for the tests
* More tests, better invalid option handling at repo level
* .read() certain amount of bytes to validate the content
* Better invalid argument handling
* Better progress bars by using .getsize() API
* For fallback upload, use the from_info.name instead of temporary naem
* introduce --jobs for add/import-url
* cli: remove: fix error message for the case when file is not under control (#4497)
* cli: remove: fix error message for the case when file is not under control (#4497)
fixes after code review
* cli: remove: fix error message for the case when file is not under control (#4497)
2nd round of fixes after code review
* cli: remove: fix error message for the case when file is not under control (#4497)
3rd round of fixes after code review
* pygit2: implement get/set/remove refs
* pygit: implement iter_refs
* pygit: implement resolve_rev
* pygit2: implement working tree checkout
* pygit2: implement reset, explicitly read/write index when used
* pygit2: checkout does not need to reset index
* pygit2: handle "-" branch shorthand in checkout
* tests: install post-checkout hook should be run via gitpython
* exp apply: use stricter exception handling for merge conflict
* pygit: fix index read/write usage
* gitpython: handle untracked file conflicts in stash apply
* fix flaky/race index issue
* tests: upgrade azurite and enable loose mode
Azure tests started failing recently. They still work with real azure, so this is on azurite side.
* tests: use cloud instead of workspace
also gdrive doesn't support it yet :(
* dvc repro: fail on checkpoint stages
* exp res: check for moved HEAD/baseline before resuming
* exp run/res: always consider last checkpoint as applied for workspace runs
Looking at the log is confusing, as the version is stuck in `1.11.0` whereas the recent one is `1.11.10`.
As the updater is also dependent on that version, it prints an ugly update message while running too.
https://github.com/iterative/dvc/issues/5023#issuecomment-758450519
* scm: implement gitignore; use `pygit2` backend
* Add is_ignored implementation on `pygit2` and `gitpython` backends.
* Also implement `._reset()` on `repo.scm` which is most of the changes.
* fix tests
* use _backend_func
* minor fixups
* S3Tree._list_paths now recognizes the max_items limit
* Use modern approach + add a test
* refactor tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* lint
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Moved exception handling imports wherever they occur
* Move dvc.tree and dvc.external_repo imports to it's
nearest use, as it is expensive for `--version`.
* Move analytics import to when it's used
* Move format_link to place where error occurs
A great benefit to this is that, if the dvc.yaml file
is never interpolated, the parser is never built, which
reduces the gap even further between 1.0 and 2.0 files
* Fix partial file checkouts in the same directory
* unittest => pytest
* Try to rationalize changed() to take filter_info
* Use path.read_text() to verify the directory structure
* Try to use specialized hash, fall back to general one if doesn't exist
* Cover subdirectories in hash specialization
* coverage: add a test for empty subdirectories
* Move out the dir_info filtering block
* hash_origin => path_info
* cleanup
* Move hashing functionality after the existence checks
* Simplify it a bit
* Update dvc/cache/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions from code review
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions
* simplify tests
* change filenames
* Some visual changes for consistency
Co-authored-by: Ruslan Kuprieiev <[email protected]>
I had to keep providing default value before when profiling
simple functions as an API. I added a default now, and
also made `debugtools` customizable so that you could
edit the code to, for example, have `pdb` enabled by default.
* experiments: use relpaths when tracking changed files
* experiments: suppress SCM hints in tempdir executors
* experiments: move multiprocessing error logging into child
* exp run: clarify help text for --temp
* Add pyinstrument: a statistical profiler
It is easier than cprofile, in that it can show stacktrace
of where things are slow.
Also refactored/moved debugging related utils to a separate
module
* use ipdb first, then fallback to pdb
* use ipdb first, then fallback to pdb
* plots: allow dir plots
* first scenario
* separate dvclive tests
* rename index to step
* another test
* refactor
* its ok to have step in data
* sort plots
* introduce --logs
* add logs to run commands
* variable name refactoring
* fix logs option
* fixup
* cmd: run: fix unit tests
* plots: modify: fix tests
* improve dir handling
* fixup
* dvclive: logs summary
* api: introduce logs summarization method
* handle no repo case
* fix granular dir plots
* api: dvclive: simplify
* dvclive: roll back logs flag
* cmd: introduce logs
* api: adjust to logs
* test: dvclive fix
* rename logs to dvclive
* live: command: add uninitialized
* outs: dvclive type
* dvclive: support providing config via evn variables
* dvclive: treat as metrics
* dvclive integration: add support for summary on/off
* dvc integration: add temporary test for configuration export
* remove requirements
* cmd: introduce logs
* rename logs to dvclive
* cmd: introduce logs
* remove logs leftowvers
* dvclive: env: adjust to chekpoints env variables
* command: live: remove unnecessary help
* fix linter suggestions
* type hinting
* review refactor
* replace dvclive with live
* stage: loading: move live loading to separate method
* stage: serialize: allow only single live output
* live: add show/diff
* stage: serialize: roll back single live enforcing
* fixup
* rebase fixups
* output: live: force single instance
* live: command: add unit tests
* Update dvc/output/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/stage/__init__.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/stage/serialize.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* review v2
* live: fix summary tracking by experiments
* naming fixups
* api: live: add tests
* tests: api: separate live tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* cli: config: show merged config with --show-origin
Builds on #5126 and #5184 by showing the full merged config when
listing or getting config options.
References #5126
References #5184
References iterative/dvc.org#2028Fixes#5119
* Add additional tests.
* Add test for merged config.
* config: rename methods
Co-authored-by: OLOLO ALALA <[email protected]>
* cli: config: add support for --show-origin
Support the --show-origin option for config, which, similar to git,
prefixes each config option with the source file it originated from.
Fixes#5119
* Reformat with black.
* Simplify logic for local config files.
* Add test for --show-config output.
* Update tests/func/test_config.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* stage: support explicit allow_missing in checkout
* exp: explicitly clear EXEC_BRANCH and checkout with allow_missing on
starting experiment run
* git: wrap stash apply errors with SCMError
* git: add `git status` to backends
* exp: reset staged git changes before workspace exp runs to avoid merge
conflicts
* test untracked (and staged) pipeline file and deps for both workspace
and tmpdir runs
* experiments: explicitly track pipeline files and uncached deps for exp
commmits
* fix exp typing errors
* dvc/tree/local: fix it for no repo/cache case
fixes#5069
* Update dvc/output/base.py
* Update dvc/output/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* exp apply: show passed exp name in CLI output instead of resolved git sha
* exp res: warn that --name will be ignored when resuming without
modifications
* exp gc: don't re-use gc parameter check message
* exp: update help strings
* exp: warn if exp dir contains untracked files after tmpdir execution
* #4578 added is_user_executable to dvc file
* Addressed PR feedback and introduced test_add_executable
Also fixed all tests in test_add.py
* Added a test to checkout executable file
* Pop isexec in dvc/output/__init__.py:loadd_from
* Implement test that file is executable after checkout
* Address PR feedback
* Address PR feedback
* Changed BaseTree.isexec to use path_info as an argument
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* fix formatting
* add: preserve exec
* checkout: fix setting exec
* tests: refactor checkout exec test
* tests: fix windows cases
* tests: add: fix exec test on windows
* Update dvc/tree/local.py
* Update dvc/output/base.py
* Update dvc/output/base.py
* Update dvc/cache/base.py
* Update tests/func/test_add.py
* tests: add: skip whole exec test on windows
* save isexec to lockfiles
* fix formatting
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* experiments: move executor into a module
* experiments: separate executor into base/local
* rename LocalExecutor to TempDirExecutor, run in .dvc/tmp by default
* git: restore HEAD using reset not checkout in detached_head()
* experiments: support calling BaseExecutor.reproduce directly for
workspace runs
* experiments: run single experiments in local workspace by default
* update experiments tests for workspace changes
* experiments: update checkpoints for workspace runs
* update checkpoints tests
* update test_show
* add force param for apply
* exp apply: add --force option
* exp run/res: add --temp option to force tempdir execution for single
experiments
* force apply in non-apply specific tests
* Cleanup log messages when loading stages
1. Simplified callback stage debug message
2. Reduced no lock entry message to trace
If lockfile does not have some entry, it will simply complain
that lockfile is not uptodate once.
If there is no lockfile or empty lockfile, it will complain that
lockfile not found.
3. Annoying parametrization related wdir-interpolated debug message
removed.
* Update dvc/stage/__init__.py
* black
* Skip paramdependency for vars
Previously, `configs` which was a list of PathInfo was being used
to check if the `vars` should be skipped or not.
But, the file is stored as a string that made that check redundant
which might have caused #5117 as they were being stored twice,
once as a expanded paramdependency and next as a flattened dict
from vars, which later during `flatten`, it might have panicked
from that duplication.
* Use defaultdict
* change add_vars to collect_vars
Currently, we iterate over targets manually in CLI, which results in a
complicated code and a DAG collection/check overhead. With this PR we
collect the DAG once, figure out all the needed targets within it and
just reproduce them in one pass.
* fix making too many ssh connections when pushing run cache (#4732)
* reformat
* generator should exit on GeneratorExit exception
* Update dvc/tree/ssh/__init__.py
* pool: close on Exception and not BaseException
Co-authored-by: Tomasz Łakota <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
minor refactors, code comments
Rename interim to definitions
provide location of entry during init for definition
small fixups to set_temporarily
Refactor loop to use dict comprehension
Provide same interface for definitions
Make it work for composite lists
Check for sequence rather than a list
refactor tests
Fixes#4955Fixes#4957Fixes#4963
Currently we don't handle bad name well:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: unexpected error - not enough values to unpack (expected 2, got 1)
```
With this PR, we check name to conform to either `remote.name.option` or
`section.option`, so we'll give a more friendly:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: argument name: name argument should look like remote.name.option or section.option
usage: dvc config [-h] [--global | --system | --local] [-q | -v] [-u] [-l] [name] [value]
Get or set config options.
Documentation: <https://man.dvc.org/config>
positional arguments:
name Option name (remote.name.option or section.option).
value Option value.
optional arguments:
-h, --help show this help message and exit
--global Use global config.
--system Use system config.
--local Use local config.
-q, --quiet Be quiet.
-v, --verbose Be verbose.
-u, --unset Unset option.
-l, --list List all defined config values.
```
We try to optimize `tree.exists` calls and probably few others
in that they either look directly into the workspace or,
to the cache without running graph checks. It does not seem
to be possible just to run graph checks on `find_outs_by_path`
due to those optimizations.
So, that's why, the `collect` also does a graph check for this
reason.
Fixes#5027Fixes#4010
* experiments: add utils.exp_commits for iterating over experiment commits
* repo: add support for all_experiments in brancher and used_cache
* git: make all_commits return commits in standard refs instead of --all
* gc: add --all-experiments to preserve cache for experiments
* update gc tests
* repo: support pushing and fetching specific revs internally
* exp push: push cache for experiment by default
* exp pull: pull cache for experiment by default
* fix empty git repo bug
* [WIP] git: use dulwich for add/commit
* dulwich: add is_tracked implementation
* experiments: try to workaround windows cleanup() bug on py < 3.8
* tests: experiments: convert CRLF to LF before comparing params
* trie: updated message for OverlappingOutputsError
the message given by the exception was not verbose enough why this exception is thrown.
now gives the user more information as to why the overlapping paths should be changed.
* test_run_single_stage: fixed test TestShouldRaiseOnOverlappingOutputPaths
the test is now updated to reflect the exception message changes.
Signed-off-by: metalglove <[email protected]>
* git: remove scm from backends, make backends configurable
* git: implement get/set/remove ref in gitpython
* git: support reflog --rewrite for stash
* tests: support testing individual backends in unit/scm/test_git
* stash: create reflog if it does not exist for gitpython backend
* git: fix Stash.push args typing
* experiments: handle case where args file was inadvertently committed to
git
* add test for args file being committed to git
* run/repro: show command on dry-run, add ">" to script output
This rollbacks the earlier commit to not show command during
the dry-run, and also prepends ">" in the command script
instead of "$" sign.
* fix tests
* Allow list type in STAGE_DEFINITION scheme and join(cmd) if list
* Remove join in Stage.__init__ and update test
* Loop over commands in cmd_run
* Improve logging of stage.run with detailed by command
* Fix test based on log message
* Update command logging with >
* Add tests for multi lines command
* Update > to $ in stage command log to prevent confusion with > operator
* Fix PR review about test: single quote and read_text
* tests: unify/clarify tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* WIP fixes#4621
* changes to all_branch if condition
* flatten_dict changes
* fix one test
* minor changes
* minor changes
* add precision
* tests and added precision to arguments
Co-authored-by: indh8717 <[email protected]>
2020-12-02 19:46:48 +02:00
590 changed files with 40319 additions and 24099 deletions
Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.
Example: `repro: doesn't detect input changes`
-->
## Description
<!--
A clear and concise description of what the bug is.
-->
### Reproduce
<!--
Step list of how to reproduce the bug
-->
<!--
Example:
1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->
### Expected
<!--
A clear and concise description of what you expect to happen.
-->
### Environment information
<!--
This is required to ensure that we can reproduce the bug.
-->
**Output of `dvc doctor`:**
```console
$ dvc version
$ dvc doctor
```
**Additional Information (if any):**
<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.
If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
#. Data Engineering tools such as `AirFlow <https://airflow.apache.org/>`,
`Luigi <https://github.com/spotify/luigi>`, and others - in DVC data,
model and ML pipelines represent a single ML project focused on data
scientists' experience. Data engineering tools orchestrate multiple data
projects and focus on efficient execution. A DVC project can be used from
existing data pipelines as a single execution step.
#.`Git-annex <https://git-annex.branchable.com/>`_ - DVC uses the idea of storing the content of large files (which should
not be in a Git repository) in a local key-value store, and uses file hardlinks/symlinks instead of
copying/duplicating files.
#.`Git-LFS <https://git-lfs.github.com/>`_ - DVC is compatible with any remote storage (S3, Google Cloud, Azure, SSH,
etc). DVC also uses reflinks or hardlinks to avoid copy operations on checkouts; thus handling large data files
much more efficiently.
#.`Git-LFS <https://git-lfs.github.com/>`_ - DVC is compatible with many
remote storage services (S3, Google Cloud, Azure, SSH, etc). DVC also
uses reflinks or hardlinks to avoid copy operations on checkouts; thus
handling large data files much more efficiently.
#.*Makefile* (and analogues including ad-hoc scripts) - DVC tracks dependencies (in a directed acyclic graph).
#. Makefile (and analogues including ad-hoc scripts) - DVC tracks
dependencies (in a directed acyclic graph).
#.`Workflow Management Systems <https://en.wikipedia.org/wiki/Workflow_management_system>`_ - DVC is a workflow
management system designed specifically to manage machine learning experiments. DVC is built on top of Git.
#.`DAGsHub <https://dagshub.com/>`_ - This is a Github equivalent for DVC. Pushing Git+DVC based repositories to DAGsHub will produce in a high level project dashboard; including DVC pipelines and metrics visualizations, as well as links to any DVC-managed files present in cloud storage.
#.`DAGsHub <https://dagshub.com/>`_ - online service to host DVC
projects. It provides a useful UI around DVC repositories and integrates
other tools.
#.`DVC Studio <https://studio.iterative.ai/>`_ - official online
platform for DVC projects. It can be used to manage data and models, run
and track experiments, and visualize and share results. Also, it
integrates with `CML (CI/CD for ML) <https://cml.dev/>` for training
models in the cloud or Kubernetes.
Contributing
============
@@ -199,39 +222,9 @@ Contributing
|Maintainability| |Donate|
Contributions are welcome! Please see our `Contributing Guide <https://dvc.org/doc/user-guide/contributing/core>`_ for more
Barrak, A., Eghan, E.E. and Adams, B. `On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects <https://mcis.cs.queensu.ca/publications/2021/saner.pdf>`_ , in Proceedings of the 28th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2021. Hawaii, USA.
Description: Git for data scientists - manage your code and data together
Data Version Control or DVC is an open-source tool for data science and machine learning projects.
.
Key features:
.
1. Simple command line Git-like experience. Does not require installing and maintaining any databases. Does not depend on any proprietary online services.
.
2. Management and versioning of datasets and machine learning models. Data is saved in S3, Google cloud, Azure, Alibaba cloud, SSH server, HDFS, or even local HDD RAID.
.
3. Makes projects reproducible and shareable; helping to answer questions about how a model was built.
.
4. Helps manage experiments with Git tags/branches and metrics tracking.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Peter Rowlands (변기호)
Saugat Pachhai
Matthew Beckers
eggqq007andZhenyu Gao
Batuhan Taskaya
Paweł Redzyński
Michael Lingelbach
Guillaume COTER
pwoolvett
Amey Kulkarni
whysage
Nikita Kodenko
Amey Kulkarni
Hongbo Miao
HimanshuandJorge Orpinel
Arjun Vikram
Maxime RischardandRuslan Kuprieiev
Batuhan Taskaya
Christian Clauss
Sayanta Banerjee
kareemjanoandpre-commit-ci[bot]
Hadas Arikandpre-commit-ci[bot]
Kurian Benoyandpre-commit-ci[bot]
Miguel Méndez
Tomasz Pietruszka
Jon Crall
yehiyam
Laurens Duijvesteijn
Dmitry PetrovandIvan Shcheklein
tomrutterandTom Rutter
rgvanwesepandRobert Van Wesep
ciaochiaociaoandChiao-Wei Hsu
Jan
Naveen Sinha
Bhagawan Subedi
Alex Lopez
Alex Ford
Helio MachadoandPeter Rowlands
Laurens Duijvesteijn
Corentin Hembise
Sami Jawhar
Pavel Ditenbir
RogerandPeter Rowlands
Piotr Bielak
Timofey Yefimovandkarajan1001
Asif Saif UddinandRuslan Kuprieiev
Dmitry Kropachev
Yihao JiangandSaugat Pachhai
mbiesekandRuslan Kuprieiev
Srinivas Reddy Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి)
Michael StegemanandOLOLO ALALA
Ioana Grigoropol
devramx
João Rafael
Sandeep MistryandRuslan Kuprieiev
Johnny Chen
metalglove
OrangeCubeandGary Yuan
Victor
Clément WalterandRuslan Kuprieiev
jwokatyandRuslan Kuprieiev
alex lustenberg
Indhu Kumarandindh8717