Hoping to fix `dvc[s3]` installations in pipenv. Pipenv's resolver is in loop, backtracking ruamel.yaml from 0.17.17 to 0.17.11 continuously. I think this has to do
more with `s3`'s dependency than `ruamel.yaml` as it only fails here. And, we are specifying dependency in 1.0.1 whereas aiobotocore has released 2.0 versions which is what is being installed, so it's redundant anyway. I am hopeful that this may help resolver with the simplification in constraints.
Adds new argument `fill_value` to ParallelCoordinatesRenderer.
For scalar columns, prevent converting to categorical by explictly inserting None when value is missing (value == fill_value).
For categorical columns, replace `fill_value` with `"Missing"` and ensure it is the last item so it gets rendered on top of the column.
- partially refactor existing pidfile behavior into using proc manager
- old pidfiles are now json-serialized executor info files placed in
proc manager subdirs (alongside proc manager pidfile/logs/etc)
- move multiprocessing based (attached) execution into separate
execution path from future "detached" execution support
- introduce ExecutorManager classes
- move executor init + execution from experiments.__init__ into the
manager classes
- make executors serializable via ExecutorInfo
related to #5489
Current dvc/command/experiments.py is too long for switch between cmd class
and the argument parser.
Split the sub command in dvc/command/experiments.py to individual files
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
We didn't mock `os.kill` in `test_dead_process` and the pid it
kills might coincidentlly be the same with some real process.
1. Add a mocker to mock out `os.kill` in `test_dead_process`.
2. Use reset_mock to replace setting it again.
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.
This PR also removes some redundant/broken remote tests.
fix: #7062
1. Add `send_signal`, `kill`, `terminate` to processManager.
2. Add tests for each of them.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
This is a very early version of test utils for dvc plugins. This module
consists of existing tests with minimal changes and is going to evolve
a lot as our plugins are getting filled in and pytest-test-utils matures.
This commit removes dvc specific message hardcoded in
scm.detach_head. Now that we are decoupling scm, we are
adding `client=` kwarg to be set by the caller.
Also adds a wrapper for DVC to use as GitFileSystem in dvc.fs.scm.
This is temporary, and will be renamed back to dvc.fs.git after the
split.
Note that this filesystem is not fully compliant, it's only
meant to be compatible with our previous implementation.
This removes dependency from dvc.utils and dvc.types from DVC.
Some utils like fix_env, is_binary have been reimplemented.
Regarding fix_env and is_binary, I think eventually this
should be a responsibility of DVC itself to set.
relpath implementation in DVC also works with paths in
different drives in Windows. I have changed this to use
os.path.relpath for now.
This adds a support for reporting progress from Git in a structured
format. The gitpython already provides somewhat structured progress
reporting, dulwich on the other hand only provides git-like line
messages.
As gitpython uses git binary, it already parse this line format
in a structured way. This PR makes dulwich use the same machinery
to unify progress reporting for both dulwich and gitpython python.
This does add a dependency on gitpython even on dulwich backend,
so I am iffy if we should do this.
The alternative is to translate gitpython's progress reports to
string, and just use string messages as progress reports on both
dulwich and gitpython.
Sometimes, `tmp_dir` fixtures take longer than a second. Using `func_only`
makes it only consider duration of test function run, excluding fixtures
duration.
The `exp init` tests may just keep waiting on the prompts blocking tests.
The blocking of prompts is likely a failure in tests setup or a bug
in the implementation. It's hard to find out these failures or avoid it.
If they take longer than 1 second, they most likely have failed as
most of test takes 0.1 second at max.
pygit2 wheels should be now available.
Had to skip pytest-docker for Windows + Python 3.10 due to
a dependency issue with it's dependency, see:
https://github.com/docker/docker-py/issues/2902.
Because our machine name now is the key value of the machine config
dict. And the machine config value will been overwritten by it when the
config is loaded. We can remove the field `name` from the config.
Borrowed this idea from pandas api. Current dropna methods will drop a
column or row if any data is missing. But it is also quite useful to
drop those columns or rows only if all of its elements are missing. This
is mostly used for pruning a table.
- allow mismatched RSA signature algo names in initial
SSH_MSG_USERAUTH_PK_OK response
- unit test overridden asyncssh publickey auth compatibility with
github ssh server
- vendor should use ssh-agent or default IdentityFile(s) when no SSH key
has been provided by the user
- encrypted (fallback) keys will always be passed to ssh-agent
fix: #6940
`--global/system/project/local` help message are in some irrelevant
command `status`, `ssh`, `create`, `destroy`. These commands are more
related to running instances rather than `machine config`, level
options didn't work for them.
1. Remove `--global/system/project/local` options in `status`, `ssh`,
`create`, `destroy`.
2. Add a test for it.
3. Correct a typo error.
fix: #6838
1. make machine status name optional.
2. show all machine status if no name provide.
3. UI changes for this cmd.
4. add a new test for it.
Co-authored-by: Dave Berenbaum <[email protected]>
All three of those use more or less the same logic for testing different
link types and maybe then using them. So now instead of having 3
implementations, we'll have just one.
Filesystem `transfer` now will not try to catch particular errno's, but
instead will just ignore any OSError and move down to the next link type
to try out, and if the last one fails - it will raise the last exception,
so if it is a common problem - we'll be able to see it.
This patch also removes `odb.confirmed_cache_type`, as it was caching
link type that is not really a cache-level thing, but rather a checkout thing,
that should only be cached for particular objects we are linking, as multiple
objects could be linked to different filesystems from the same odb (e.g.
external local output on hdd and local data on your ssd).
Next step here will be to introduce metadata diff, so that we can get rid of
temporary Link wrapper and introduce Transfer and corresponding transfer
config, that will know what links to use and could have some other things
like callbacks, so we don't continue passing dozens of kwargs along the stack.
Continuing from #6963Fixes#6980
After #6814 the header of `diff_table` is `a_rev/`b_rev` instead of `Old`/`New` so the name of the option was counterintuitive.
In addition, make `a_rev` show by default, for consistency with output format of other `diff_table` uses (`dvc metrics diff` / `dvc params diff`).
It'll give a error message to the user and reprompt them.
This mechanism can be extended to other prompts, and we
can show a warning message and/or reprompt them based on
validations.
Closes#6865. Part of #6446.
This method mimics pandas `drop_duplicates` when axis=rows.
For axis=cols this method will drop columns with 0-variance (only
1 unique value). NaNs (`_fill_value`) won't be counted as unique values.
So the following column (`-` would be the `_fill_value`) will be dropped:
foo
-
foo
foo
Intermediate PR in removing path_info from our code base.
It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.
For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).
Fixes#6774
It comes with the support for no. of parameters that matches with json.dumps.
See: https://github.com/willmcgugan/rich/pull/1547 and
https://github.com/willmcgugan/rich/pull/1644.
Also added support for printing json to stderr. Before we used to use json.dumps
and rich.json.JSON based on isatty. Now we use rich.print_json on both
tty and non-tty conditions, which may avoid different behaviours in the
future.
Looks like azure-core is dependent on a dependency `chardet` that aiohttp brought it with, which has been replaced with `charset_normalizer`.
But azure-core depends on internal of aiohttp and looks like it tries to set it's own encoding parser, `chardet` which is now not available.
Writing at the end of each step (create_summary) can mess up with some ML Framework progress bars, I moved the message to the monitor task, as the path won't change during training.
Fixesiterative/dvclive#177
We were previously using relpath in *all* cases, mainly to support WSL but relpath doesn't work
on macOS.
Added a check for using relpath *only* for WSL.
Fixes#6092
Currently whether or not link confirmation will run depends on
particular filesystem and existing workspace link type, which introduces
inconsistency in behaviour, where we sometimes catch unsupported links
before touching user data and sometimes we don't.
Depending on a filesyste (e.g. on NTFS), `_remove` might reset
read-only permissions in order to delete a hardlink to protected object,
which will also reset it for the object itself, making it unprotected,
so we need to protect it back.
Currently it depends on the whole `info()` output, which has too much
information that doesn't really matter. To check if file contents didn't
change mtime, size, inode combo is enough.
* executor: move local specific git initialization out of base
* machine: return kwargs for executor init instead of executor object
* executor: add initial ssh executor
* ssh-executor: disable strict host checking for dvc-machine instances
* initial tests for ssh executor
* drop GIT_SSH_COMMAND usage
* push: accept destination odb as a parameter
* separate cache initialization into executor.init_cache
* add test for ssh executor cache init
* remove dead code
* machine: fixture-ize default test scenarios
* test SSHExecutor.from_machine
* tpi: return default SSH port for constructing executors
* iterate on ui, has tree and shows to-be-generated yaml content
* order fs tree structure alphabetically
* always show the heading
* hide dvclive on tree if not on live mode
* Change message at the end
* s/skip/omit
* add link on the header
* remove newline on non-interactive mode
* Add experiment name check.
1. Add experiment name check (https://git-scm.com/docs/git-check-ref-format)
2. Add duplicate exp name check.
3. Add some unit test for it.
* Ban slash / in dvc exp names
* Use dulwich backend for ref name checking
* Some bug fix
* Update dvc/repo/experiments/__init__.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/repo/experiments/__init__.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some review changes
* Make some funtion more reusable.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
If supported, reflink, will provide the fastest transfer possible.
That being said, right now we use `move` in most of the places,
which still takes presidence.
Pre-requisite for #6387
scm.install does 3 different things, and all of those are
very much tied to DVC. So, as we are splitting scm out,
it should now belong to `dvc.install`.
Linux/Windows machines have only 2 cores, whereas macOS machines have 3 cores. Maybe we are stretching available resources by 2x, let's see if `-n=auto` improves test freezes.
* git: support client kwargs for auth in remote methods
* tests: use docker for git/ssh server
* move dulwich backend into subdir
* add base sync<->async support
* dulwich: add initial asyncssh SSH vendor
* dulwich: use asyncssh as the default SSH vendor
* add test for git over ssh
* test read (fetch) over ssh
* ci: re-enable ssh tests
temporary workaround for #6788, #6789
* move async stubs into dvc.scm.asyn
* tests: use sshfs and not dvc.fs.ssh
* allow explicit 0-length read in asyncssh vendor
* `dvc machine`: add a new command `rename` to it
fix#6481
1. add new command `dvc machine rename`
2. add a unit test for the call.
3. add several functional tests for it.t push myself fix6481
* Move rename logic to tpi
* use tpi.state_mv instead of rename
* Rebase to a newer version
* Update tpi version
* Remove unused assert and try catch
* throw error if new parameters are added by dvc run exp -S
Throws an error if dvc run exp --set-param references a parameter key path
that is not in the target config file. This is done to prevent silent errors
in experimentation when new params are accidentally added instead of modifying
current ones due to typos.
This implementation is based on initial suggestions in issue #5477.
* fix mypy confusion from overwriting a var
* tests: remove modify param test that was adding params
One of the test in test_modify_params was still adding a parameter
* tests: remove param modification during exp run in test_untracked
Spurious usage of modifying a parameter when the params.yaml is not yet existing in test_untracked.
* remove benedict dependency for verifying params
Instead, the keypath util module from benedict is adapted into the dvc codebased and used
natively to verify that a keypath exists within a dict.
* exp: list all invalid param modifications in error message
* exp-run: more specific exception for new params via --set-param
Uses established MissingPramsError instead of DvcException. Modified message to be more similar to other uses of this exception.
* Revert "remove benedict dependency for verifying params"
This reverts commit 6170f16dcc4033667fbacb0d0bc13c60898f58c4.
* exp: use internal benedict import for validating params
* exp: combine new param validation into merge_params
Moves new param validation into existing merge_params function as an optional check
* exp: minor ammendments to error message
* Revert "tests: remove param modification during exp run in test_untracked"
This reverts commit c9eb1a62af58586509ded362cc79e75ddc4b119c.
* exp: fix test_untracked to not fail due to new params
Fixed by creating the params file with the new parameter already present.
* machine: Add a new command `dvc machine status`
fix#6482
1. add a new command `dvc machien status`
2. add a unit test for the cli call.
3. add a unit test for the shown result.
* Update dvc/command/machine.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Update dvc/command/machine.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Update machine.py
* Remove sensitive fields
* Update dvc/machine/__init__.py
Co-authored-by: Casper da Costa-Luis <[email protected]>
* Update tests/func/machine/__init__.py
Co-authored-by: Casper da Costa-Luis <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use white list to limit shown fields
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Casper da Costa-Luis <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* dvc: drop 3.6 support
Considering that 3.6 has ~2 months until EOL, and our deps are
dropping support for it already (e.g. pygit2), we are droppipping
it as well. This will reduce the load on our test CI as well, since
we've started supporting 3.10 recently.
* setup.cfg: adjust requirements
* daemon: remove 3.6 logic
* tests: utils: remove pre 3.7 logic
* use setuptools_scm and build_requires
* remove kwargs from setup.py
* fix tests; fetch in depth in github actions
* use build to build sdist/wheel
* try updating pip and setuptools
* update pip
* remove or unpin wheel
* bring back the version in pyinstaller
* Update scripts/build.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* advertise Python 3.10 support
Pygit2 does not have windows wheels yet and no pyarrow
for 3.10. But these platform-dependent things and
remote-dependent things should not stop us from
advertising support as we are pure-python based
package and we cannot claim support for all of our
dependencies.
* set pygit2 for Python 3.6 to <1.7
Dependabot was throwing Illformed requirements `"< '3.10'"` for the
hdfs. But I noticed that it did accept other requirements where we
do similar. So it seems the indentation fixes it.
Even then, dependabot did not update any dependencies. I removed
setup.py file, after which immediately it started creating PRs.
So, I recknoed that it must have been due to some mechanism
in place from dependabot to disallow execution, so I tried to
remove few blocks of code from setup.py. And, I was eventually
able to figure out that it was from the block where we import version.
So, I added a try-except block and it magically again worked. :)
Let's see how it goes.
1. update diff.py,experiments.py,metrics.py,params.py,status.py to
print prettify json
2. rewrite write_json in __init__.py to accept default function
3. add new unit test for write_json in tests/unit/ui/test_console.py
4. Fix#6712
Co-authored-by: Zhenyu Gao <[email protected]>
* doctor: fixes unknown cache director error on some windows mapped network shares
On Windows, resolve calls os.path._getfinalpathname, which calls GetFinalPathNameByHandle.
This starts from the final resolved path of the file in the NT namespace (e.g. \Device\Mup\server\share...)
and works backward as required to get the VOLUME_NAME_DOS. If the final path is a UNC share (i.e. \Device\Mup),
We fix this by calling resolve() also on the partition mountpoints
We also take the opportunity to make checks in case a PermissionError is returned to us
Fixes github.com/iterative/dvc/issues/6305
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* doctor: fixes unknown cache director error on some windows mapped network shares
On Windows, resolve calls os.path._getfinalpathname, which calls GetFinalPathNameByHandle.
This starts from the final resolved path of the file in the NT namespace (e.g. \Device\Mup\server\share...)
and works backward as required to get the VOLUME_NAME_DOS. If the final path is a UNC share (i.e. \Device\Mup),
We fix this by calling resolve() also on the partition mountpoints
We also take the opportunity to make checks in case a PermissionError is returned to us
Fixes#6305
Co-authored-by: CHETOUAN <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: CHETOUAN <[email protected]>
* exp init: introduce `--with={default,live}` flag
This flag will select the different mode of outputs of stage.
With `--with=live`, it'll enable `live` setting in the stage.
On interactive mode, it'll always ask user if they want to
use live (unless `--with=live` is specified), and then it'll
skip metrics/plots question and moves on to ask for path to
dvclive path.
Also, if live setting is enabled in the stage, it'll set the
`models` to be a checkpoint output.
* s/with/type and don't ask for live interactively
1. update ls __init__.py to output pretty JSON format
2. introduce write_json function into ui __init__.py
Co-authored-by: Zhenyu Gao <[email protected]>
This was used before on opening http urls as a stream,
but we no longer use those (as we use fsspec's open method).
One thing to note however is that, compared to previous
implementation, fsspec's http filesystem's open is not very
robust to failures.
speedcopy only patches shutil.copyfile, which we don't even use
anywhere because it lacks callbacks/progress and we instead use
our own copyfile implementation, so this is totally useless for
us and can be removed.
```
2021-09-21T00:15:31.6789059Z ____________________ ERROR at setup of test_get_url[azure] _____________________
2021-09-21T00:15:31.6790443Z [gw0] linux -- Python 3.7.12 /opt/hostedtoolcache/Python/3.7.12/x64/bin/python
2021-09-21T00:15:31.6791202Z
2021-09-21T00:15:31.6792291Z request = <FixtureRequest for <Function test_get_url[azure]>>
2021-09-21T00:15:31.6793561Z
2021-09-21T00:15:31.6794120Z def fill(request):
2021-09-21T00:15:31.6794792Z item = request._pyfuncitem
2021-09-21T00:15:31.6795627Z fixturenames = getattr(item, "fixturenames", None)
2021-09-21T00:15:31.6797090Z if fixturenames is None:
2021-09-21T00:15:31.6797922Z fixturenames = request.fixturenames
2021-09-21T00:15:31.6798497Z
2021-09-21T00:15:31.6799393Z if hasattr(item, 'callspec'):
2021-09-21T00:15:31.6800256Z for param, val in sorted_by_dependency(item.callspec.params, fixturenames):
2021-09-21T00:15:31.6801309Z if val is not None and is_lazy_fixture(val):
2021-09-21T00:15:31.6802233Z > item.callspec.params[param] = request.getfixturevalue(val.name)
2021-09-21T00:15:31.6802912Z
2021-09-21T00:15:31.6803937Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pytest_lazyfixture.py:35:
2021-09-21T00:15:31.6804952Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-09-21T00:15:31.6806085Z /home/runner/work/dvc/dvc/tests/remotes/azure.py:102: in azure
2021-09-21T00:15:31.6807090Z container.get_container_properties()
2021-09-21T00:15:31.6808748Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/core/tracing/decorator.py:83: in wrapper_use_tracer
2021-09-21T00:15:31.6822551Z return func(*args, **kwargs)
2021-09-21T00:15:31.6824143Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/storage/blob/_container_client.py:497: in get_container_properties
2021-09-21T00:15:31.6825038Z process_storage_error(error)
2021-09-21T00:15:31.6825652Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-09-21T00:15:31.6825908Z
2021-09-21T00:15:31.6826916Z storage_error = HttpResponseError("Operation returned an invalid status 'The value for one of the HTTP headers is not in the correct format.'")
2021-09-21T00:15:31.6827586Z
2021-09-21T00:15:31.6828351Z def process_storage_error(storage_error): # pylint:disable=too-many-statements
2021-09-21T00:15:31.6829086Z raise_error = HttpResponseError
2021-09-21T00:15:31.6829597Z serialized = False
2021-09-21T00:15:31.6830066Z if not storage_error.response:
2021-09-21T00:15:31.6830554Z raise storage_error
2021-09-21T00:15:31.6831558Z # If it is one of those three then it has been serialized prior by the generated layer.
2021-09-21T00:15:31.6832480Z if isinstance(storage_error, (PartialBatchErrorException,
2021-09-21T00:15:31.6833636Z ClientAuthenticationError, ResourceNotFoundError, ResourceExistsError)):
2021-09-21T00:15:31.6834687Z serialized = True
2021-09-21T00:15:31.6835753Z error_code = storage_error.response.headers.get('x-ms-error-code')
2021-09-21T00:15:31.6836524Z error_message = storage_error.message
2021-09-21T00:15:31.6837001Z additional_data = {}
2021-09-21T00:15:31.6837398Z error_dict = {}
2021-09-21T00:15:31.6837739Z try:
2021-09-21T00:15:31.6838506Z error_body = ContentDecodePolicy.deserialize_from_http_generics(storage_error.response)
2021-09-21T00:15:31.6839345Z # If it is an XML response
2021-09-21T00:15:31.6839837Z if isinstance(error_body, Element):
2021-09-21T00:15:31.6840285Z error_dict = {
2021-09-21T00:15:31.6840750Z child.tag.lower(): child.text
2021-09-21T00:15:31.6841435Z for child in error_body
2021-09-21T00:15:31.6841822Z }
2021-09-21T00:15:31.6842195Z # If it is a JSON response
2021-09-21T00:15:31.6842692Z elif isinstance(error_body, dict):
2021-09-21T00:15:31.6843413Z error_dict = error_body.get('error', {})
2021-09-21T00:15:31.6843893Z elif not error_code:
2021-09-21T00:15:31.6844347Z _LOGGER.warning(
2021-09-21T00:15:31.6845611Z 'Unexpected return type % from ContentDecodePolicy.deserialize_from_http_generics.', type(error_body))
2021-09-21T00:15:31.6846887Z error_dict = {'message': str(error_body)}
2021-09-21T00:15:31.6847294Z
2021-09-21T00:15:31.6847730Z # If we extracted from a Json or XML response
2021-09-21T00:15:31.6848205Z if error_dict:
2021-09-21T00:15:31.6848792Z error_code = error_dict.get('code')
2021-09-21T00:15:31.6849704Z error_message = error_dict.get('message')
2021-09-21T00:15:31.6850578Z additional_data = {k: v for k, v in error_dict.items() if k not in {'code', 'message'}}
2021-09-21T00:15:31.6851392Z except DecodeError:
2021-09-21T00:15:31.6851801Z pass
2021-09-21T00:15:31.6852290Z
2021-09-21T00:15:31.6852601Z try:
2021-09-21T00:15:31.6853156Z # This check would be unnecessary if we have already serialized the error
2021-09-21T00:15:31.6853821Z if error_code and not serialized:
2021-09-21T00:15:31.6854423Z error_code = StorageErrorCode(error_code)
2021-09-21T00:15:31.6855164Z if error_code in [StorageErrorCode.condition_not_met,
2021-09-21T00:15:31.6856229Z StorageErrorCode.blob_overwritten]:
2021-09-21T00:15:31.6856938Z raise_error = ResourceModifiedError
2021-09-21T00:15:31.6857739Z if error_code in [StorageErrorCode.invalid_authentication_info,
2021-09-21T00:15:31.6858641Z StorageErrorCode.authentication_failed]:
2021-09-21T00:15:31.6859468Z raise_error = ClientAuthenticationError
2021-09-21T00:15:31.6860256Z if error_code in [StorageErrorCode.resource_not_found,
2021-09-21T00:15:31.6861256Z StorageErrorCode.cannot_verify_copy_source,
2021-09-21T00:15:31.6861998Z StorageErrorCode.blob_not_found,
2021-09-21T00:15:31.6862716Z StorageErrorCode.queue_not_found,
2021-09-21T00:15:31.6863718Z StorageErrorCode.container_not_found,
2021-09-21T00:15:31.6864526Z StorageErrorCode.parent_not_found,
2021-09-21T00:15:31.6865432Z StorageErrorCode.share_not_found]:
2021-09-21T00:15:31.6866549Z raise_error = ResourceNotFoundError
2021-09-21T00:15:31.6867609Z if error_code in [StorageErrorCode.account_already_exists,
2021-09-21T00:15:31.6868499Z StorageErrorCode.account_being_created,
2021-09-21T00:15:31.6869383Z StorageErrorCode.resource_already_exists,
2021-09-21T00:15:31.6870728Z StorageErrorCode.resource_type_mismatch,
2021-09-21T00:15:31.6871688Z StorageErrorCode.blob_already_exists,
2021-09-21T00:15:31.6872429Z StorageErrorCode.queue_already_exists,
2021-09-21T00:15:31.6873252Z StorageErrorCode.container_already_exists,
2021-09-21T00:15:31.6874124Z StorageErrorCode.container_being_deleted,
2021-09-21T00:15:31.6874922Z StorageErrorCode.queue_being_deleted,
2021-09-21T00:15:31.6875857Z StorageErrorCode.share_already_exists,
2021-09-21T00:15:31.6876608Z StorageErrorCode.share_being_deleted]:
2021-09-21T00:15:31.6877289Z raise_error = ResourceExistsError
2021-09-21T00:15:31.6877807Z except ValueError:
2021-09-21T00:15:31.6878259Z # Got an unknown error code
2021-09-21T00:15:31.6878648Z pass
2021-09-21T00:15:31.6878942Z
2021-09-21T00:15:31.6879424Z # Error message should include all the error properties
2021-09-21T00:15:31.6879908Z try:
2021-09-21T00:15:31.6880416Z error_message += "\nErrorCode:{}".format(error_code.value)
2021-09-21T00:15:31.6881289Z except AttributeError:
2021-09-21T00:15:31.6883082Z error_message += "\nErrorCode:{}".format(error_code)
2021-09-21T00:15:31.6883807Z for name, info in additional_data.items():
2021-09-21T00:15:31.6884446Z error_message += "\n{}:{}".format(name, info)
2021-09-21T00:15:31.6884932Z
2021-09-21T00:15:31.6885741Z # No need to create an instance if it has already been serialized by the generated layer
2021-09-21T00:15:31.6886538Z if serialized:
2021-09-21T00:15:31.6887019Z storage_error.message = error_message
2021-09-21T00:15:31.6887526Z error = storage_error
2021-09-21T00:15:31.6887908Z else:
2021-09-21T00:15:31.6888499Z error = raise_error(message=error_message, response=storage_error.response)
2021-09-21T00:15:31.6889380Z # Ensure these properties are stored in the error instance as well (not just the error message)
2021-09-21T00:15:31.6890078Z error.error_code = error_code
2021-09-21T00:15:31.6890630Z error.additional_info = additional_data
2021-09-21T00:15:31.6891956Z # error.args is what's surfaced on the traceback - show error message in all cases
2021-09-21T00:15:31.6892672Z error.args = (error.message,)
2021-09-21T00:15:31.6893112Z try:
2021-09-21T00:15:31.6893784Z # `from None` prevents us from double printing the exception (suppresses generated layer error context)
2021-09-21T00:15:31.6894873Z > exec("raise error from None") # pylint: disable=exec-used # nosec
2021-09-21T00:15:31.6896126Z E azure.core.exceptions.HttpResponseError: The value for one of the HTTP headers is not in the correct format.
2021-09-21T00:15:31.6897385Z E RequestId:d83b8102-bde1-4a77-aa7a-b67bbc2c4a80
2021-09-21T00:15:31.6898080Z E Time:2021-09-21T00:09:53.815Z
2021-09-21T00:15:31.6898578Z E ErrorCode:InvalidHeaderValue
2021-09-21T00:15:31.6899259Z E headername:x-ms-version
2021-09-21T00:15:31.6900066Z E headervalue:2020-10-02
2021-09-21T00:15:31.6900355Z
2021-09-21T00:15:31.6901781Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/storage/blob/_shared/response_handlers.py:177: HttpResponseError
```
* rebased PR 6111 onto master
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use make_tmp_dir fixture in tests
Co-authored-by: U119511 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Alexander Meier <[email protected]>
Co-authored-by: meierale <[email protected]>
* tests: additional callbacks test for remaining filesystems
* requirements: bump pydrive2 to 1.9.4
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Trying to see if getting rid of this dependency, our issue with timeout
gets solved or not.
We use timeout-minutes anyway on GHA, so we probably don't need it.
Locally, users can monitor this themselves.
* machine: add `machine list`&`machine modify`
1. add new command `dvc machine list`
2. complete `dvc machine modify`
3. add new unit tests for this two commands to ensure the call.
4. add functionality tests for two commands
* Update dvc/config_schema.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update tests/func/machine/test_machine_config.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some reviewed problems
1. capitalize global variables.
2. remove precision restriction.
3. solve windows path seperator problem.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* dvc cli: fixed error in 'dvc config --list' in non repo dir
Fixes#6496
* Issue 6496: Added test that covers the issue
* dvc cli: fixes error in 'dvc config <name>' in non dvc repo dir
Fixes the specified error. Did some refactoring based on previous commits.
Also added a test case for setting config in non dvc repo dir.
Fixes#6496
* Issue-6496: dvc config --<level> and dvc config --list --<level> will now throw config error if ran in non dvc repo with repo only level
* Issue 6496: Segregated config levels into repo and system levels that helped in simplifying the logic to decide appropriate level at the time of cli operation.
The current ci failure was caused by wrong order of the experiments.
Because our test script runs so fast that they might share the same
commit timestamp, that makes them in an uncertain order. And then gives
a wrong branch_base.
* output: use remote kwarg instead of odb
Fixes#6596
* Update dvc/output.py
Co-authored-by: Dave Berenbaum <[email protected]>
* add test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Dave Berenbaum <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fs.download: add callback support
* fix lint issues
* compat for gdrive, use fs to upload text for tests
* fix typing issues
* skip gdrive test
* Update tests/func/test_fs.py
* fs.upload: add support for optional callback
Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Show csv format of experiments
fix#5989 and #5446
1. add --show-csv to dvc exp show
2. add a new functional test for csv format
3. add a new unit test for show_experiements
4. solve name duplication in table common headers (Created,Parent, etc... ), metrics files and param files.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Set timeout for jobs in Github Actions
Sometimes tests are getting frozen(most likely an issue with `pytest-xdist`), and sometimes the pip resolver is slow due to some weird constraints set by us or one of our dependencies.
The default timeout is 360 minutes (6hrs), but our test completes on ~18 minutes at maximum, so I think 30mins should be enough to account for occasional CI slowdown.
Also keeping the total job timeout at 45 mins, we usually require just 4 mins of extra time to do installation and setup but giving it 11 more minutes just in case.
Similarly, `lint` completes in <5 mins usually, but giving it a 10 mins timeout.
This should give us early feedback on the issue.
* Update setup.cfg
* Update setup.cfg
* machine: install dvc via startup_script by default
* machine: support configuring instance type and startup script
* accept startup_script as a config relpath
* use pushd/popd in default install script
* extract transfer related logic from odb to fs.utils.transfer
* don't makedirs on dirinfo
As local odb needs to set a mode for dir, we cannot create a directory
in the transfer
Making sure that script takes a bit more time than the await time
I am not really sure this was the cause though.
Also the tests are flaky only in MacOS, but I haven't been able to reproduce.
* Clean up remotes's exps
fix#6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail
Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
* Added dvclive to tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Moved dvclive to rest_requirements
* Update setup.py
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Added refresh tag
* Formatted
* Added refresh_seconds to live
* Added refresh test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add refresh_seconds to render
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update html.py
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Update README.rst
Fixes typos and makes the related technologies stand out from their description.
* Apply suggestions from code review
* Update README.rst
* Wrapped modified portions to 80 chars or less
Left others untouched to simplify review
Co-authored-by: Jorge Orpinel <[email protected]>
* added auth to iter_url and created custom open for http fs
* added default value to auth
* Update dvc/utils/http.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/fs/http.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* pr fixes
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Remove all of the experiments in the workspace
fix#5676
1. remove all the experiments in workspace
2. add tests for it
* Only remove baseline ones
* Add a new arguments --all
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Rename and shortcut for these commands
* Solve the issue from removing a changing list.
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* delete workspace option
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Replacing DVC_ANALYTICS. This is following our other env conventions and
will not have effect if set to false/no/etc, and will only work if set
to true/yes/y/etc. Will disable analytics even if it is explicitly
enabled in our configs.
Kudos @skshetry
We've used updater for years now to notify our users ASAP about new
releases. These days dvc is more stable and we've actually been
receiving numerous complains about updater being too annoying and
even introduced a config flag to turn it off, which a lot of people do
and which defeats its purpose. Instead of introducing some complex logic
that would notify more rarely and once for the user instead of once for
every dvc project they have - this PR just turns it off for everything,
except `dvc doctor/version`, which is the command that people use when
they are running into issues and that will give them a hint to just try
out the newest version first, as it might be already fixed.
This PR is a more of a compromise, to get the most out of updater while
removing a pain point of it being too annoying, without investing much
time into it and not having to maintain complex unnecessary logic. We
might get back to introducing something more intricate in the future,
if the need arises.
Fixes#5912
* add support for Python 3.10
pyarrow is not available, so we are skipping tests for it.
And, restricting `hdfs` extras to be not installed in Python3.10.
Pathlib removed `_init` method and `init` parameter, so just working
around that. It's not a big change.
* skip windows on 3.10; update pip/wheel/setuptools during install
pywin32 is not yet available for 3.10 Python, so skipping those for now
* install libgit2 with Python3.10 as pygit2 wheel is not available
* install libgit2
* only run on macOS, libgit2-1.1 not available on ubuntu
* Remove a special queued experiments
fix#6157
1. dvc exp remove not accept queued experiments name
2. add some tests for this feature
* Extract tests and add revision support
1. Extract remove experiments to a new file.
2. revision can be used to remove special queued experiment
* Accept shortened revisions
* Split removing committed and queued exp functions
* Api name change
* Return to the old API name
* shorten some functions
* Error message change
* Better test cases, more corner case
* Still raise exception in a mixed case
* Push running checkpoint to remote
Fix#6182
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Get the branch name
* Finish this PR
1. move env name to dvc/env.py.
2. add some tests for it.
* Update after finally commit
* Some problems found in the review
1. change the behavior of self remote
2. do not use string DVC_EXP_AUTO_PUSH
3. downgrade the logger level in auto push
4. use full branch ref
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* add a remote check before auto push
* Split current one env into two
* Rename DVC_EXP_CHECKPOINT_PUSH to DVC_EXP_AUTO_PUSH
* value error
* Remove getenv_bool switch to env2bool
* Better on handling Dulwich exceptions
* Name changing removing prints
* Update dvc/env.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some changes in review
* Add validation to repo URL
* Add new type of SCMError
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Use spy tests call counts and args
* Update dvc/scm/git/backend/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/scm/git/backend/dulwich.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove validate_git_repo to validate_git_remote
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* demonstrate issue while sending analytics report
When sending analytics, we are not flushing the data to the temp file
which causes failure at times, especially on Windows
* close file before calling daemonized analytics send command
This way, the contents bufferred will get a chance to be flushed.
* Remove rwlock decorator for the run() method of the Stage class.
Instead of pre-emptively grabbing read locks on the dependencies
and write locks on the outputs when calling the run() method,
delegate the locking mechanism to other methods.
This results in more selective locking that is less likely
to cause a LockError when executing steps in parallel.
* Remove rwlocked decorator from reproduce() method of Stage class.
* Reinstate read-lock (not write) for run method of Stage class
* fix bugs and typos
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* objects.status: use object IDs instead of naive objects
* objects.transfer: use object IDs instead of naive objects
* objects: add ReferenceHashFile
* odb: add ReferenceObjectDB
* repo: collect used object IDs instead of naive objs
* support basic/naive fs serialization
* objects.stage: stage file objects in addition to trees
* objects: remove save(), update transfer()
* output/dep: update for osave removal
* path_info: support both url and path attributes
* status/gc: use object IDs instead of naive objs
* diff: update staging usage
* commit: update staging usage
* tests: clean staging when needed
* tests: update for save/transfer/stage changes
* dvcfs: don't allow config serialization (repofs should be used instead)
* utils: return mtime as str, size as int
* tests: update unit tests for save/stage/transfer changes
* objects: use Repo.open when needed in ReferenceHashFile
* fix ls test staging
* objects.stage: don't skip hash check when loading from state
- individual hashes will still be loaded from state if they are still
valid
* ReferenceHashFile: serialize repo fs paths relative to repo root
* preserve state when staging
* objects.tree: skip hash computation in digest() when we have already
loaded hash from state
* always check file existence in referencehashfile
* update repofs test since stage no longer allows nonexistent files
* odb: only protect on check() if hash was verified
* staging: use unique memfs namespace for staging per dest ODB
* tests: clean full staging namespace
* refodb: preserve repofs cache dir
* repodependency: stage imports to local ODB and strict check circular
imports
* fetch: fetch staging last
* staging: make dest odb mandatory and preserve state
* refodb: cache nonlocal fs's
* fix rebase errors
* repofs: support full instantiation from fs config
* repofs: preserve subrepo factory for erepos
* tests: clarify repofs dirty file behavior
* Fix merging of params when empty
Recent release of python-benedict==0.24.1 started ignoring empty dictionaries
during merge. The way we expected to work was questionable and also how
benedict itself tries to handle the references to empty dictionary is
also questionable.
I consider this to be a proper way to handle this even if this gets fixed
on the benedict side.
* Revert "setup: limit python-benedict version (#6377)"
This reverts commit 15de826266.
* add comment for why passing through benedict
add additional excludes to DefaultAzureCredential
which are needed in a Windows Environment to
use az cli provided credentials, etc.
Co-authored-by: Alexander Meier <[email protected]>
* objects: migrate remote status to objects.status
* data_cloud: use objects.status
* objects: migrate remote._process to objects.transfer
* data_cloud: use objects.transfer
* objects: handle memfs staging objects in transfer
* fetch/import: use unified transfer()
* update get_remote and transfer exception usage
* update tests
* objects.transfer: use src ODB verify rules after xfer
* odb: move index from remote into odb
* objects.transfer: skip src status check when possible
* update tests
* remove dvc.remote
* odb.index: migrate to diskcache
* objects: fix status/transfer optimization
* update index usage
* state: remove dead sqlite related code
* update cloud tests
* drop unnecessary remove in tests
* use abstract base class in odb index
* status: load trees from local cache instead of remote odb when possible
* handle status when dir cache is explicitly removed
* params/metrics/plots/exp show: introduce onerror
* fixup
* plots/metrics/params: removing errors
* removing more errors
* removing errors
* some more removing
* new plots ux
* new ux for metrics
* new ux for params
* fixing tests
* fixups
* minor test fixes
* minor test fixes
* commands: plots/params: fix ui imports
* working on error handlng decorator, stopped at show checkpoints
* error_handler, further dvelopment
* plots just before moving loading to collect
* appending errors to results: initial passing
* fixup
* onerror: remove onerror fixture
* fixups
* plots/metrics/params/experiments: temporarily remove error warnings
* plots: extract parsing out of class method
* metrics/experiments: provide exception serializer for show-json
* params/plots/metrics: signal errors
* rebase fixups
* fix TODOs
* fixup
* skshetry's review refactor
* Migrate to ossfs
fix 6177
* Add verification to object stored on oss
* Change the oss remote's dependency.
pass all of the tests
* Performance optimaztion and ossfs version update
* Remove the argument to set the resumable get/put
1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default
* Use a new api for performance optimization
* Without filesize progress bar version
* With progress bar version
* Bump and add comments
1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger
* Change the name of kwrags following parent class.
* Bump to new ossfs and rewrite upload_fobj
Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj
* Optimization on rm_files
* Some optimization follow the old version.
* Update dvc/fs/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/objects/db/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* bump into ossfs 2021.7.2
* Bump into ossfs 2021.7.3OC
* Switch to real_oss_server
* Switch to a real oss server
* Use the oss bucket located in US.
* Remove some emulator related code
* Add envs to ci
* remove some oss pytest XFAIL
Because we didn't use emulator, we can remove some pytest XFAILs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* exp: allow fetching from arbitrary executor by git URL
* exp: write info for running executors to .dvc/tmp/exp/run
currently only includes PID and git fetch URL
* exp: refactor stash/queue cleanup
* exp: add ability to collect dict of currently running experiments
* exp show: display run/queue state in the table
* exp run: unlock main repo while running tmpdir executors
* add tests for new exp show behavior
* fix workspace checkpoint run handling
* fix issue where running stash revs were shown as queued
* use "running"/"queued"
* add executor (location) column
* compare: add TabularData.is_empty
* use td.is_empty to drop state/executor
* use "workspace" and "temp" as executor names
* fs: use fsspecwrapper in memfs
* odb: add memfs staging ODB in ODBManager
* odb: add read_only attribute for odb instances
* objects: handle memfs paths when staging trees
* odb: remove git ODB
* output: use staging ODB
* RepoDependency: use staging odb for imports
* dvcfs: handle case where dir cache is unavailable
* fix circular import check
* odb: use temp local ODB for staging
* objects: use staging ODB in stage() and tree load
* objects: add typing for add/check/load/stage
* output/dep: use new staging
* add unit tests for local odb staging
* localfs: account for string paths in move()
* include local in odb.by_scheme
* tests: count dir cache like any other object in test_pull_git_imports
* odb: clear staging on gc()
* objects: add tree.get/filter/insert
* output: use obj.filter/get for granular dir commit
* squash filter logic output
* dependency: don't filter imports at the dep level
* don't init staging ODB for external cache
* use staging ODB when computing tree hash for external outs
* remove unneeded force in test granular commit
* use unique-per-manager memfs URLs for staging
* odb: make staging per-ODBManager
* objects: move state dependent staging into ODBManager
* objects: support checking multiple odbs
- staging dependent checks can be done via ODBManager.check
* checkout: do state lookup before explicit stage() in _changed()
* output: use odbmanager based staging
* RepoDependency: use odbmanager based staging
* update obj staging unit tests
* checkout: remove check(), handle individual file exceptions on link()
* diff: use odbmanager based staging
* objects: add ObjectPermissionError for read/write perm errors
* odb: move staging out of odbmanager
* objects: handle staging ODB in objects.stage()
* objects.tree: don't stage anything in tree methods
* odb: clear staged objects on gc()
* remote: don't filter objs by scheme, objs are already grouped by ODB
* diff: update objects.stage usage
* output/dep: update stage() usage
* update ODB tests
* update func tests
* remove memfs from FS_MAP
* revert unneeded objects.check() changes
* remove unused tree.insert()
* revert unused odbmanager changes
* add separate odb error classes
* stage external outs directly in ODB
* check for obj existence in main ODB before staging
* use .dvc/tmp/staging instead of per-odb staging
* gc: only gc loal staging once
Kudos @skshetry for discovering that we've been publishing py2.py3
wheels on pypi and researching the cause.
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
* Update s3.py
pass verify flag when creating a session
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fixed diff summary to only show states with values.
* Removed redundant comma by using join.
* Swithced for loop to generator for better readability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* update dvc exp run -h message
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* add: do not delete stage files before add
Because of the way we collect stages and cache them, we were not able to
collect them for the `add` without removing them from the workspace.
As doing so, we'd have two same/similar stages - one collected from the
workspace and the other just created from the `dvc add` in-memory.
This would raise errors during graph checks, so we started to delete them
and reset them (which is very recently, see #2886 and #3349).
By deleting the file before we even do any checks, we are making DVC
fragile, and results in data loss for the users with even simple mistakes.
This should make it more reliable and robust.
And, recently, we have started to keep state of a lot of things, that by
resetting them on each stage, we waste a lot of performance, especially
on gitignores. We cache the dulwich's IgnoreManager, which when resetted
too many times, will waste a lot of our time just collecting them again
next time (see #6227).
It's hard to say how much this improves, as this very much depends on
no. of gitignores in the repo (which can be assumed to be quite in number
for a dvc repo) and the amount of files that we are adding
(eg: `-R` adding a large directory).
On a directory with 10,000 files (in a datadet-registry repo),
creating stages on `dvc add -R` went from 64 files/sec to 1.1k files/sec.
* add tests
* make the test more specific
* output: str only once
Calling `__str__` takes a long time and if we are dealing with a large
dataset, it will result in a very substantial time. For example,
locally, for 700K objects, it took 20sec more.
Related to #6276
* output: don't use os.path.join
Takes around 2sec for 700K objects.
* output: use fs.sep
* output: remove unused sep
* separate validation and transformation of args from add func
* make tests pass for Python3.6
* cleanup
* use dict for attr-access and patch boundarguments.arguments
* Keep trailing slash when checking if directory is gitignored
* Calling is_ignored with correct type (could be Path-like)
* Fix test to comply to annotated types
* Fix separator issues
* Relaxed typing in is_ignored
* Added test
* Test scm
* Fix test on Windows
* Expanded test_ignored_dir_unignored_subdirs
* Even more assertions
* objects: add ExternalRepoFile object type
* RepoDependency: implement get_obj (use ExternalRepoFile)
* drop distinction between used external and used objects
* update tests
* remove unneeded fetch from ExternalRepoFile
* wrap erepo errors as an object error
* update cloud/remote functions for new used_objs behavior
* fix update() rev/locked behavior
* update import tests
* remove ExternalRepoFile
* move get_obj and get_used_objs implementations back into repo dependency
* return objects plus associated ODB (remote) in get_used_objs
* odb: add tmp_dir field and support creating Remote from an odb
* return dict mapping odb -> objects in get_used_objs
* use dummy git odb for git imports
* fetch: support pulling from specific odb
* update cloud funcs for get_used_objs usage
* update tests for used_objs behavior
* use fetch in dep.download
* catch circular imports from local fs repos
* add tests for chained and circular imports
* move tmp_dir into odb config
* update return docstring for repo.used_objs
* Add Studio to the related technologies list
* Add CML to DVC Studio description
* fix k8s names a bit
Co-authored-by: Ivan Shcheklein <[email protected]>
* add: prevent crash while getting relpath
Since python 3.8 os.path.realpath has resolved windows network shares
to UNC paths. This breaks the logic for relpath as the UNC and mapped
drive paths don't have an obvious common prefix. As it is not easy to go
back to the mapped drive path from UNC path, this is resolved by always
using the UNC paths for calculating relative paths in windows.
Fixes#5832
* relpath: Update changes following review
Two changes:
- use pytest.mark.skipif rather than if statement inside test to make
it clearer that test is windows specific
- Use tail rather than upper_path to denote rest of path after drive
name taken off (to match python documentation)
Pull request: #5837
* tests, linter: update to add test case and apply linter though pre-commit
Co-authored-by: Tom Rutter <[email protected]>
* objects: make HashInfo & HashFile hashable
* stage/output: collect objects (HashFiles) instead of NamedCache
* remote/data_cloud: use objs instead of NamedCache
* repo: use objects instead of NamedCache
* rerun black after rebase
* remove NamedCache usage from run-cache
* drop NamedCache
* update HashInfo.__hash__
* update unit tests
* revert remote_name change
* start cleaning up test failures
* fix tree obj expansion
* update missing cache logging
* fix external dict handling
* update naive odb.gc
* update state save behavior for pull
* update test_data_cloud
* update test_remote
* fix gc with multiple repos
* objects.tree: support filtering w/preserved hash_info
* fix stage cache restore
* drop fs.path_info from HashFile.__hash__
* objects: add optional name attribute
* revert missing cache logging name/suffix changes
* revert name changes in remote tests
* review fixes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update dvc/remote/base.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* config, remote: Made S3 CA bundle customizable
botocore allows a path to a custom CA bundle either by passing a path
to the CA bundle file into the verify argument of
boto3.session.Session.client or passing None (the default) which will
fall back to the AWS config. Previously, the DVC config only accepted a
boolean into the ssl_verify option in the remote S3 config. This changes
the DVC config to accept both string and None in addition to boolean and
defaults to None. I also changed the default for ssl_verfiy to None in
BaseS3FileSystem. Thus, if ssl_verify is not provided, botocore will
fall back to the AWS config.
Testing
Unit tests to cover the changes to the config schema and addition
ssl_verify types that will be passed into S3FileSystem. Also, ran
dvc push -r object-store data/cifar-10-python.tar.gz
in my work environment that has a private S3 endpoint that requires a
custom CA bundle, both with and without ssl_verify specified in the
config. This was successful, showing that communication could be
established. And I ran
dvc remote modify object-store ssl_verify "$HOME/.aws/cabundle.pem"
and confirmed that the custom CA bundle path was added to the config.
Fixes#6012
* Removed default None on ssl_verify
Responding to PR comment, removed the Optional, default None on
ssl_verify since the config keys are optional by default. Rather
than a missing ssl_verify producing a None that eventually gets
filtered, it doesn't appear in the parsed config in the first place.
Co-authored-by: Robert Van Wesep <[email protected]>
astroid==2.5.7 has improved inference and was throwing
errors that were false negatives before. Fixing these
and ignoring some.
Also added `funcy.decorator` to the `signature-mutators`,
as it changes function signature (i.e. provides `Call` as
a first argument which is not required to pass).
Example errors that we were getting because of this:
```
tests/remotes/gdrive.py:40:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
************* Module dvc.stage.serialize
dvc/stage/serialize.py:38:1: E1120: No value for argument 'func' in
function call (no-value-for-parameter)
************* Module dvc.fs.http
dvc/fs/http.py:19:1: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
************* Module dvc.fs.gdrive
dvc/fs/gdrive.py:66:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
dvc/fs/gdrive.py:459:5: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
```
Pre-requisite for removing fs.path_info and switching to regular strings
as paths in filesystems.
With new filesystems we no longer need the zoo of different dependency
and output types, as most of them are now just handled by the underlying
filesystem. Though deps/outs are still due for a refactor in other
aspects.
* Decoupling dvcignore and fs
[x] remove dvcignore from fs objects
[x] attach dvcignore to repo
[x] add ignore check to functions(add, checkout, output, pipeline, etc)
[x] pass all tests
[x] final check
* Solve all of the arugments error.
* add dvcignore to dvc list
* Solve some test_ignore cases
* add dvcignore to checkout.
1. add dvcignore to 'checkout', 'add'
2. add ignore_filter to fs
* Solve two tests
* Get rid of ignore_filter
* Add ignore examine to download
* Solve a test issue
* Solve dvcignore and subrepo problems
* Solve sub repo error
* Revert some changes
* pass all tests
* Remove debug codes
* Some final exams
* dvcignore trie tree only update when needed
* reuse of the dir result in path walk.
* Some change after code review.
* Recover something from false `--force` push
* add abspath trans to dvcginore
* pass pylint
* pylint related
* Update dvc/fs/repo.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Changes related to reviews.
* dvcignore call api change
from dvcignore.walk_files(fs.walk(path_info)) to dvcignore.walk_files(fs, path_info)
* Get rid of PathInfo in test_ignore.py
* fs.walk follow the arguments of os.walk
* Dvcignore as parameters
* pass dvcignore as arguments to checkouts
* Decoupling state and repo
* Update dvc/dvcfile.py
Co-authored-by: Paweł Redzyński <[email protected]>
* Update dvc/objects/stage.py
Co-authored-by: Paweł Redzyński <[email protected]>
* Solve some small problems
* Add tests to verify that dependency now not ignore
fix#5256
* Update dvc/ignore.py
* Update dvc/fs/git.py
* Update dvc/repo/__init__.py
* Some small improvements
* pass black
* Refactor is_ignored
* Update dvc/output/base.py
* repofs: use dvcignore in exists/isdir/isfile
Per https://github.com/iterative/dvc/pull/5812/files#r630604524
Co-authored-by: myname <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
We used to allow changing default output and error stream
but that was problematic for the testing. Now it just uses
sys, so capsys should work for the testing.
The proper way to change sys.stdout and sys.stderr is
by using 'redirect_stdout' now.
* try to close dvc.state on exit
I have a feeling that it is causing the issues with
shutil.copytree and some PermissionError during 'init',
as we are not closing it between multiple 'main()' calls.
But I am not quite sure, and this seems like a good thing to
do anyway.
* fixup
* close dvc on exit
* close cache on tests and on init
* azure: use a custom event loop for authentication
* add test_temp_event_loop
* wrap open() too!
* ignore aiohttp warnings for normal level use-cases
* rename tests/unit/remote/test_azure to tests/unit/fs/test_azure
we just need to mark them with needs_internet marker.
Unfortunately, it's not possible to do that automatically
with the fixtures as fixtures are per-session and we need
something on more higher level (which the marker is not passed
through in the pytest).
This is because, the repo root is used from a realpath, and the stage.wdir is abspath.
We might need to use same kinds of path, but this patch seems to be the easiest/fastest
way to fix the flaky tests (if all the tests agree).
* dvcfile: add check for gitignored when validating filenames
* update tests
* update exception using in tests
* review fixes
- clarify method name
- use FileIsGitIgnored
I tried increasing max_run to very high number: 1000. Even in those
cases, it failed 947 times. So I don't really see a reason to mark
it as flaky. Anyway, we are treating it as xfail by keeping it failing
in the CI, so this PR does not change anything for us. :)
I found that it's harder to style a header with the list using index,
they are to be applied by header name. Otherwise, we might end up using
a different header for coloring, or worse throw some error because the
header name and style don't match.
Row does have it's property of index, so it makes sense there.
* apply: allow applying exp with multiple possible branches
* default to new checkpoint branch after applying ambiguous commit
* output shas in addition to exp names by default
* update show tests
* git: allow force checkout in detach_head context
* exp run: error when trying to resume from a non-head checkpoint without
modifications
* add test for apply/resume from intermediate checkpoint
* list some possible heads when multiple matches
* use new table data structure for diff and show
* fix params diff showing changes
* metrics show: use default precision
* skip precision format for exp diff params
* exp diff: separate tables in --quiet mode
* change precision to signify significants digits
changed f specifier to g, so that we show significant digits
instead of fractional digits. This does changes how precision
worked for metrics/params diff/show.
* stage,commit: add support for only committing data stages from
experiments
* exp run: `dvc commit` data deps when stashing exp changes
* add test for modified DVC-tracked data when generating experiment
* test with and without params change
* handle run without target
* objects: merge transfer() with stage()
* objects: drop streamfile
* objects: preserve the signature of get_file_hash
* objects: override hash_jobs if jobs= passed
* objects: support no progress bar directive
* tests: spy on _get_file_hash
* objects: update=... directive
* update: only transfer missing caches with --to-remote
* tests: remove local_remote from the primary test (due to it moves)
* objects: add move=True/False option
* objects: implement _get_file_obj on top of get_file_hash
* objects: use threading on save()
* objects: pass jobs= on to-cache as well
* objects: use in place tqdm
* objects: check for FileExistsError
* Update dvc/objects/__init__.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Due to https://about.codecov.io/security-update/ , there is a potential that we've leaked these. This PR takes a more granular approach when setting secret env vars so that this doesn't happen in the future.
We'll change these secrets after this PR is merged.
* exp: add util method for getting baseline HEAD during run
* exp: use baseline for dvc commands which compare to HEAD during exp run
* add test for new util method
* update plots diff test mock
* Add support for kerberos token based auth
* Added kerb_ticket as attribute to HDFSFileSystem
* Lint fix for config schema
* Updated black style changes
* Removed staticmethod decorator from hdfs
This part of logic used to be only used by external repo for cloned
repos, but since we now handle local repos without cloning, we don't
always pass scm instance. This PR automatically creates scm instance
for such cases.
Fixes#5590
* params: read from ParamsDependency if possible
* params: introduce include_untracked flag
* params: include untracked upon targets
* experiments: add option to show tracked params
* fixing test
* params: rename tracked to deps
* params: roll back to default params.yaml behavior
* experiments: add option to show only used params on diff
* Update dvc/command/experiments.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/experiments.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/params.py
Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
* exp show: Allow for wildcard patterns in include and exclude params/metrics
Fixes#5642
* Add test case for nested metrics/params
* match full string in _filter_name
* pre-commit
* Update dvc/command/experiments.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Added SSL verification option
* Changed param name to avoid conflicts
* verify_ssl now accepts strings
* Added simple unit tests
* Changed parameter name fron verify_ssl to ssl_verify so it's consistent with HTTP remotes
Forget to change 1 test
* add processing on ssl_verify param
typo
* revert back to bool param
* Update dvc/config_schema.py
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
This is achieved by using `lock_repo`
context manager that can be used in a recursive depth
that only resets stages when it has left any lock_repo
blocks.
Also added some ui related tests for the status which I found
missing
Before, we used to set it during initialization. But this
made it hard to test using capsys as capsys changes the
reference to sys.stdout and sys.stderr with the mocked one
during the test. But since we are using dvc.ui globally and
setting it to a real sys.stdout and sys.stderr before capsys
comes into effect, testing was not possible with it.
Now, by using property hack, we will be using the values of
sys.stdout and sys.stderr during the time of write, so the mocking
would work and would be easier to test.
* dvc: fix breakage with rich==10.0.0
Change in the arguments of __rich_measure__.
Fixes#5722
* dvc: Fix linting breakage from rich==10.0.0
Fixes: #5733
* dvc: set lower bound for rich dependency to 10.0.0
Fixes#5722
#. Data Engineering tools such as `AirFlow <https://airflow.apache.org/>`,
`Luigi <https://github.com/spotify/luigi>`, and others - in DVC data,
model and ML pipelines represent a single ML project focused on data
scientists' experience. Data engineering tools orchestrate multiple data
projects and focus on efficient execution. A DVC project can be used from
existing data pipelines as a single execution step.
#.`Git-annex <https://git-annex.branchable.com/>`_ - DVC uses the idea of storing the content of large files (which should
not be in a Git repository) in a local key-value store, and uses file hardlinks/symlinks instead of
copying/duplicating files.
#.`Git-LFS <https://git-lfs.github.com/>`_ - DVC is compatible with any remote storage (S3, Google Cloud, Azure, SSH,
etc). DVC also uses reflinks or hardlinks to avoid copy operations on checkouts; thus handling large data files
much more efficiently.
#.`Git-LFS <https://git-lfs.github.com/>`_ - DVC is compatible with many
remote storage services (S3, Google Cloud, Azure, SSH, etc). DVC also
uses reflinks or hardlinks to avoid copy operations on checkouts; thus
handling large data files much more efficiently.
#.*Makefile* (and analogues including ad-hoc scripts) - DVC tracks dependencies (in a directed acyclic graph).
#. Makefile (and analogues including ad-hoc scripts) - DVC tracks
dependencies (in a directed acyclic graph).
#.`Workflow Management Systems <https://en.wikipedia.org/wiki/Workflow_management_system>`_ - DVC is a workflow
management system designed specifically to manage machine learning experiments. DVC is built on top of Git.
#.`DAGsHub <https://dagshub.com/>`_ - This is a Github equivalent for DVC. Pushing Git+DVC based repositories to DAGsHub will produce in a high level project dashboard; including DVC pipelines and metrics visualizations, as well as links to any DVC-managed files present in cloud storage.
#.`DAGsHub <https://dagshub.com/>`_ - online service to host DVC
projects. It provides a useful UI around DVC repositories and integrates
other tools.
#.`DVC Studio <https://studio.iterative.ai/>`_ - official online
platform for DVC projects. It can be used to manage data and models, run
and track experiments, and visualize and share results. Also, it
integrates with `CML (CI/CD for ML) <https://cml.dev/>` for training
models in the cloud or Kubernetes.
Contributing
============
@@ -199,7 +222,9 @@ Contributing
|Maintainability| |Donate|
Contributions are welcome! Please see our `Contributing Guide <https://dvc.org/doc/user-guide/contributing/core>`_ for more
details.
details. Thanks to all our contributors!
|Contribs|
Mailing List
============
@@ -229,11 +254,7 @@ Barrak, A., Eghan, E.E. and Adams, B. `On the Co-evolution of ML Pipelines and S
Description: Git for data scientists - manage your code and data together
Data Version Control or DVC is an open-source tool for data science and machine learning projects.
.
Key features:
.
1. Simple command line Git-like experience. Does not require installing and maintaining any databases. Does not depend on any proprietary online services.
.
2. Management and versioning of datasets and machine learning models. Data is saved in S3, Google cloud, Azure, Alibaba cloud, SSH server, HDFS, or even local HDD RAID.
.
3. Makes projects reproducible and shareable; helping to answer questions about how a model was built.
.
4. Helps manage experiments with Git tags/branches and metrics tracking.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Saugat Pachhai (सौगात)
dependabot[bot]
David de la Iglesia Castro
Paweł Redzyński
Ruslan Kuprieiev
Peter Rowlands
karajan1001andPeter Rowlands
pre-commit-ci[bot]
dberenbaum
Kwon-Young ChoiandPeter Rowlands
Gustavo José de Sousa
Dimitar Todorov
Димитър Тодоров (Dimitar Todorov)andSaugat Pachhai
Batuhan Taskaya
Guðmundur HeimissonandRuslan Kuprieiev
Ruslan Kuprieiev
Matthew Beckers
eggqq007andZhenyu Gao
Michael Lingelbach
Guillaume COTER
pwoolvett
Amey Kulkarni
whysage
Nikita Kodenko
Amey Kulkarni
Hongbo Miao
HimanshuandJorge Orpinel
Arjun Vikram
Maxime RischardandRuslan Kuprieiev
Batuhan Taskaya
Christian Clauss
Sayanta Banerjee
kareemjanoandpre-commit-ci[bot]
Hadas Arikandpre-commit-ci[bot]
Kurian Benoyandpre-commit-ci[bot]
Miguel Méndez
Tomasz Pietruszka
Jon Crall
yehiyam
Laurens Duijvesteijn
Dmitry PetrovandIvan Shcheklein
tomrutterandTom Rutter
rgvanwesepandRobert Van Wesep
ciaochiaociaoandChiao-Wei Hsu
Jan
Naveen Sinha
Bhagawan Subedi
Alex Lopez