`--untracked-files` and `--unchanged` coupled with `--granular`
puts a lot of stress on `show_status`. Currently, with those options,
dvc spends ~26s on `_show_status` function.
Unhappy with the above result, I looked into improving the situation with
`console.out`. It did reduce that to 9s but it still felt like a lot.
Then I dropped to good old colorama, which dropped to 3s(term)/1.24(redirected).
I tested with buffering output which dropped this further to
~600ms(term)/~100ms(redirected).
This may seem like premature optimization, but this was preventing me
from figuring out perf issues in granular outputs, and I believe the
user should not be penalize for using feature that we already offer.
`dvc <command> | head` for example could fail with SIGPIPE and
would print a traceback with BrokenPipeError exception. Now we
redirect do /dev/null in case of error as suggested in Python's
docs: https://docs.python.org/3/library/signal.html\#note-on-sigpipe.
fix: #8031
Currently when we use tmp directory to run a experiment, we set
`EXEC_HEAD` , `EXEC_BASELINE` and `EXEC_MERGE` heads locally but forget
to remove it. This cause some problem when we use `exp show`
1. Clean `EXEC_HEAD` , `EXEC_BASELINE` and `EXEC_MERGE` after we pushed
them to tmp_dir
2. Add a unit test for it.
Co-authored-by:
Followed from #7592
1. Seperate remove method to a new file.
2. Add queued, failed, processed flags to remove.
3. Add new unit tests for queue status --queue/fail/success
4. Implement methods to remove done tasks.
5. add some new unit and functional test for celery_queue.remove
6. bump into dvc-task version 0.0.13
fix: #7950
Currently queue worker status are not show in `queue status` makes it
hard to tell if the worker is still living.
1. add a new method `active_worker` to return the currently running
worker node name.
2. give every worker a unique name.
3. `queue status` will show how much worker running at present.
Currently, all of our celery workers are called `dvc-exp-worker` and
they will write to the same log path.
We give every single worker a unique name and seperate they logs if
under a debug log level
fix: #7840
Now queue remove can not accept short rev as name.
1. Refactor the queue entry match in `queue remove`
2. Better output if no queue task found for name.
3. short rev on kill can only be work for rev with lens longer than 4
fix: #7590
1. Add two new command `queue status` and `queue attach`
2. Add unit test for the CLI
3. Make `Running` exps can be found by `get_queue_entry_by_names`
1. Add some more unit test for `celery_queue.kill`
2. Implement `kill` argument for `celery_queue.shutdown`
3. Use dataclass's internal methods instead of `__eq__`
fix: #7591
1. Add a new command `dvc queue`
2. Add two sub-command `dvc queue remove` and `dvc queue kill`
3. Add a unit test to test them
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
fix: #7587
1. Add implement kill method for local queue class
2. Add a test to make sure the following job will be success after the
original job was killed.
3. Some refactoring work on `exp remove`
1. Implement `--time` to `exp gc`
2. add a unit test for it.
3. Rename `--time` to `--date` and change the warning msg.
4. To get a better msg for `gc` and `exp gc`
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: Dave Berenbaum <[email protected]>
1. Add argument `commit_time` to `gc`/`brancher` and `iter_revs`
2. Add unit test for iter_revs.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
fix#678
1. Add a new flag `--time` to `dvc gc`.
2. Modify the unit test of `dvc gc`
3. Fix an argument checking error and update the error msg.
Co-authored-by: Dave Berenbaum <[email protected]>
Allow to use dictionaries as values for template interpolation but only inside the `cmd` key.
See tests/func/parsing/test_interpolated_entry.py::test_cmd_dict for detailed syntax.
Add `config.parsing` section for configuring behavior of ambiguous data types
like booleans and lists.
Closes#6507
Uses `repo.params.show` with custom error_handler and postprocess the outputs for more user-friendly structure.
Extend `repo.params.show` to accept `stages` argument to cover the "params of current stage" use case.
Currently, there is no way to find run-cache in HTTP remotes.
In the future, we could use directory-listings from supported
webservers to find files.
Temporarily, we are restricting the run-cache uploads. This is a bit
strict, we could have only restricted the downloads, but it might be
more confusing to the users, so I went to categorically restrict
stage cache uploads and downloads for http filesystems.
We were passing paths with '\\' in the filename instead of posix paths to
remote filesystems which was causing failures to upload.
Although in some filesystems, this was failing silently, so upload would seem
to have passed, but the download later would fail.
`fetch_all_exp()` in `clone()` was called with `url="origin"`, which
resulted in the operation being performed with the remote URL defined
in the cloned repo's config, which did not include any credentials
initially provided to clone.
Fixes#7670
Use relpath to repo.root_dir as keys in deps and outs.
Add `use_cache` and `is_data_source` for outs.
This allows to differentiate git-tracked dependencies and
intermediate outputs.
Closes#7575Closes#7790
This was working for LocalFileSystem as they work for any directories/path.
It's hard to test this these days, as we don't have remote tests inside
dvc, so it was not caught.
Potentially fixes#7718.
The tests should be in dvc-objects repo npw, and will be migrated soon.
The nanotime and diskcache are specified in dvc-objects, so we don't need
those here.
This moves get_fs_config/get_fs_cloud and ODBManager to DVC
as they require DVC related utilities, and we don't need them
strictly in dvc-objects/dvc-data.
This also adds support to ReferenceHashFile to load from
any filesystem, using it's module name and filename.
Now we can get rid of DVCFileSystem dependency in
ReferenceHashFile.
This is a temporary solution, we really need a registry.
(I am not quite sure if this works in pyinstaller mode,
but I think it should, it's how fsspec does it if I understand
correctly.)
transfer raises internal TransferError, merge raises internal MergeError
and local ODB raises ODBError on unprotect.
transfer's error is reraised to FileTransferError back in DataCloud.
This commit:
* defines a `Ignore` protocol in `dvc.objects`, that expects a `find` method.
This will be used in dvc-data and dvc-objects instead of depending on dvc.ignore.
dvcignore is expected to respect this protocol, which it does.
* renames `dvcignore` kwarg to `ignore` in `dvc-data` and `dvc-objects`.
* teaches dvc.cli that `IgnoreInCollectedError` is a known exception
and is logged correctly and adds a test for the behaviour.
I am not happy that we need to hardcode `.dvcignore` in dvc-data
but I don't see any easy way, since we always try to avoid
staging if the tree contains .dvcignore file, even if .dvcignore
is not passed.
* Introduce ConfigError and AuthError in dvc.objects.fs and raise it
instead of dvc.config.ConfigError.
* Provide hints and links by catching those errors in dvc.cli.main,
i.e decouple UI messages from filesystems for AuthError and
ConfigError.
* Copy flatten/unflatten utils to dvc.objects.fs.utils.
Now HashedStreamReader will be used to calculate md5 in fobj_md5/hash_file.
Also:
* adds in `bytes_md5` that may be useful for testing.
* adds a hasher-registry with hashlib
* adds a `text=` kwarg to be able to specify to use dos2unix or not
(useful for testing)
* makes `localfs` the default filesystem when `fs` arg is not passed
(makes it easier to test in the REPL)
Note that this will change the behaviour of DVC to always show
prompt even in --quiet mode, which is what we need.
Previously it'd prompt without any message.
It turns out that diskcache.Disk saves passed arguments to the database
and uses that all the time. So this means that we are not forward
compatible as it breaks moving from newer version of DVC to an older
one. We don't promise this, but this breaks our benchmarks which
are probably not isolated enough.
Regarding the use of cache, it is a bit iffy to extend it this way.
We only need this type for error messages. We could consider removing
this, but since the error message is documented and it's not too hard
to extend this :), I decided to preserve the message.
This commit moves `HashedStreamReader`/`get_hash_file`/`file_md5`
into `dvc.objects.hash`.
The `get_hash_file` has been renamed to `hash_file`.
This also copies `relpath` and `is_exec` to the `dvc.fs.utils`.
When staging a directory, always save a "raw dir object" to the odb.
If the corresponding ".dir" object has not been added to the odb,
`stage()` calls can load the tree from the raw dir object instead of
rebuilding it by walking the directory.
This can lead to significant speed improvements when calling
`dvc status` for modified directories.
Fixes#7390
In the spirit of fsspec.generic, it is for cross-filesystem operations.
Trying to avoid `fs.utils` as much as possible, but we have some
functions that are really utilities coming, so I am making a place
for them. :)
Current naming for `DvcFileSystem` and `RepoFileSystem` is confusing, because
dvc is not only about data and it is much more natural to expect something
named `dvcfs` to include complete view for your whole repo (including git
files), which is what `repofs` is doing right now.
`DvcFileSystem` can provide a view not only for the repo data, but also for
external data (e.g. local external outputs). So it makes more sense to call
it a `DataFileSystem`, since it really represents the data management side
of dvc.
Our gitfs paths are currently tied to the local path in which the cloned
repository resides in, which results in a lot of hacky assumptions throughout
the code.
This PR detaches gitfs from local paths completely, making dvc work with a
proper fsspec gitfs implementation.
Part of #7543
Tqdm supports __bool__, so we should really use `is not None` test
rather than boolean check. This is not used anywhere right now, but
I'd prefer being correct.
This makes dvcfs/repofs much easier to work with and allows it to support
isabs/abspath/etc correctly, as previously any full path (with root_marker "")
would be impossible to distinguish from a relative one.
This also detaches external workspaces from the main repo workspace in
DvcFileSystem, which was necessary to be able to distinguish dvcfs abspaths
from local external output paths. Note that currently we still have limited
support for those (e.g. get/api supports them, but import doesn't) and this
change keeps it that way.
fix: #7448
Current we skip on those experiments which already existed on both sides, but
do not exclude them from the result list. And if we have diverged experiment,
pulling/pushing them will stop the current progress and raise an exception, but
didn't give any information on the condition of other experiments.
1. Change the exp pull and push to match the new API in `scmrepo`.
2. Handle sync status after `exp pull/push` finished.
3. Bump scmrepo to 0.0.19
Co-authored-by:
There is no support for chunk_size kwarg in fs.open.
Also, `upload_fobj` is now able to figure out proper
chunk size from fileobj, so I don't think we need this on _upload_file.
Before, as we were not passing `total`, the Tqdm progress bar would not
show the bar or not show any progress on the bar. As we get that value lazily,
we should force it to always show the bar.
Also updated the progress bar to say which repo is being cloned.
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.
The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.
This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.
The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.
This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
It is not completely fsspec compliant, but it's upto the level
required by our FSSpecWrapper.
Next thing to tackle would be to merging `fs.base.FileSystem`
and `fs.fsspec_wrapper.FSSpecWrapper` into one filesystem
that is lazily-loaded and is itself fsspec-compliant.
Also adds a kwarg `cancel_on_error` on the constructor to support
cancelling futures on `shutdown`. This is not done by default to
be backward compatible with `concurrent.futures.ThreadPool` executor.
`_validate_config()` had a bug: it would not take into account that Google
Drive credentials can also be passed using `GDRIVE_CREDENTIALS_DATA` as
documented here [1].
This patches the validation logic to also accept this environment variable as a
means to provide service account credentials.
Before, commands like `dvc push` would exit with:
```
ERROR: failed to push data to the cloud - To use service account, set
`gdrive_service_account_json_file_path`, and
optionally`gdrive_service_account_user_email` in DVC config
Learn more at <https://man.dvc.org/remote/modify>
```
[1]: https://dvc.org/doc/user-guide/setup-google-drive-remote#authorization
Co-authored-by: Laurens Duijvesteijn <[email protected]>
Bumps [mypy](https://github.com/python/mypy) from 0.931 to 0.942.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.931...v0.942)
---
updated-dependencies:
- dependency-name: mypy
dependency-type: direct:development
update-type: version-update:semver-minor
...
Bump mypy from 0.931 to 0.942
after mypy 0.940, it disabled all class scoped imports because it will
cause crashes.
1. upgrade mypy from 0.931 to 0.942
2. Solve the error `class scoped import` in mypy 0.942
This removes callback from internal APIs: `_list_paths`, `list_hashes`,
and `_hashes_with_limit`. These are all iterators and hence can just be
replaced by wrapping a progressbar over those iterators in the caller
side.
We still have progressbars in `hashes_exist`, `_estimate_remote_size`,
`_list_hashes_traverse` and `list_hashes_exists`, which should gradually
lifted up and replaced.
Webdav filesystem expects path to be relative
to the base url, instead of having full prefix.
So we are setting it to "" (root) through `_strip_protocol`
which converts fully-qualified name to the file-system
specific path.
fix: #7449
`exp remove`: should show the experiments that were removed
1. experiment.remove now returns the experiment list been removed.
2. modify the functional test to these the returned list.
3. use ui to show the removed list.
Co-authored-by: Dave Berenbaum <[email protected]>
fix: #7155
1. rename flags `-A/--all-commits` in `exp remove`
2. add new flag `-n/--num` in `exp remove`
3. unify the revision collection in `exp remove`
4. add unit and func tests for `exp remove`
Co-authored-by: Jorge Orpinel <[email protected]>
fix: #7154
1. rename flags `-A/--all-commits` in the `exp pull/push`
2. add new flag "--rev" and "--num" in the `exp pull/push`
3. Unify the collection of revs in `exp push/pull`
4. add unit and func tests for `exp push/pull`
fix: #7153
Unify the flags in `exp list` to `exp show`
1. Add new `-A`, `-n` flags to `exp list` command.
2. Add unit test for the flag parse.
3. unify the revs retrive logic to `exp show`.
4. Add functional test for `exp list`
5. Update the documents.
Co-authored-by: Jorge Orpinel <[email protected]>
- Moves workspace tree for interactive mode after prompts. Previously
this was displayed before the prompts.
- Remove the yaml contents and the confirmation prompt in an interactive
mode.
- Simplify `init_interactive()`.
fix: #7329
In a previous PR, we introduce a bug which use `HEAD` as the workspace
commit, while if there is a checkpoint is running, the `HEAD` is
actually changing all the time, we should use
`refs/exps/exec/EXEC_BASELINE` instead.
1. add `fix_exp_head` transfer in `dvc/scm.py::resolve_rev`
2. remove fix_exp_head from `diff`, `metrics diff`, `params diff` and
`plots diff`
3. modify a functional test to simulate the running conditions
Currently we use both absolute and relative (depend on cwd) paths, which is
terrible for an api, as their interpretation depends on external factors.
This PR makes dvcfs accept only paths relative to the repo root for regular
outputs and absolute paths for external outputs.
Pre-requisite for making repofs use same relative paths.
Introducing simple repo tree that combines all workspaces into one structure,
and allows us to detach data representation from pipelines.
Pre-requisite for dvc-data extraction and using relpaths in dvcfs/repofs.
Using outs directly breaks isolation and makes it really hard to understand
what's going on.
Redundant abspaths will go away in followups when moving to relpaths.
metadata was introduced before our migration to fsspec and no longer needed
now. This is a pre-requisite to using data index in dvcfs/repofs.
Note that a bunch of awkward flags like isdvc/isout/etc are still kept, and
will be removed in followups.
With our staging process automatically capturing file metadata, we no longer
need to set isexec ourselves.
In followup PRs we will also use meta during checkout, so that we won't need
to use Output.set_exec by hand.
Removed `--include-metrics` / `--include-params` and
`--exclude-metrics` / `--exclude-params`.
Removed `--no-timestamp` . Can be done by `--drop Created`.
`--drop` and `--keep` operate directly on the table columns.
`--keep` does not perform any filtering. It's only used
to specify columns to keep despite the other filtering.
For example `--only-changed --keep foo` will prevent
`foo` from being removed by `--only-changed`.
Another example, `--drop train.* --keep train.dropout`
will remove all columns matching `train.*` except for
`train.dropout`.
Closes#7079Closes#7080
Pre-requisite for #6434
By splitting dvc.cli into a module, we'll be able to share cli-related
utilities, split parser-related stuffs into smaller parts, improve
reusability and improve API for command extensions in the future.
Before it used to say 'failed to reproduce dvc.yaml', which is now changed
to print 'failed to reproduce train' or something similar.
For stages that are in dvc.yaml in the current working directory, it will
just print the stage names. For stages in other dvc.yaml files, it'll print
in <relative_path>:<stage_name>.
fix: #7196
New feature: now `dvc exp pull` can accept a list of experiment names as input.
2. add support to list input for the `exp pull`
3. add function tests for the multi exp pull
4. add unit test for the arg parse
fix: #7196
New feature: now `dvc exp push` can accept a list of experiment names as input.
1. refactor of experiment push
2. add support to list input for the `exp push`
3. add function tests for the multi exp push
4. add unit test for the arg parse
- prerequisite for remote execution
- allow explicitly pushing imports which would normally be skipped
(internal-use only, does not apply to `dvc push`)
fix: #7152
1. Add `--rev` flag to `dvc exp show`.
2. Add -1 support for `--num` flag`.
3. Extract revision found logic to `utils`.
4. Brancher use this revision found logic.
5. Update command unit test
6. Add a util unit test.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Rich added `vertical` kwarg in `add_column`, which linter
is now complaining (as it should). But this may keep on breaking
when rich adds/removes args/kwargs in the future.
So, this commit uses (*args, **kwargs) instead of specifying particular
parameters explicitly. This also sets `collapse` directly to the
`rich.table.Column` rather than extending through our one `Column`
implementation.
mypy:
```
dvc/utils/table.py: note: In member "add_column" of class "Table":
dvc/utils/table.py:24:5: error: Argument 7 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['top'], Literal['middle'], Literal['bottom']]" [override]
def add_column( # pylint: disable=arguments-differ
^
dvc/utils/table.py:24:5: note: This violates the Liskov substitution principle
dvc/utils/table.py:24:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
dvc/utils/table.py:24:5: error: Argument 8 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['fold'], Literal['crop'], Literal['ellipsis'], Literal['ignore']]"
[override]
def add_column( # pylint: disable=arguments-differ
^
dvc/utils/table.py:24:5: error: Argument 12 of "add_column" is incompatible
with supertype "Table"; supertype defines the argument type as "Optional[int]"
[override]
def add_column( # pylint: disable=arguments-differ
^
Found 3 errors in 1 file (checked 129 source files)
dvc/ui/table.py:87: note: In module imported here,
dvc/compare.py:25: note: ... from here:
dvc/utils/table.py: note: In member "add_column" of class "Table":
dvc/utils/table.py:24:5: error: Argument 7 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['top'], Literal['middle'], Literal['bottom']]" [override]
def add_column( # pylint: disable=arguments-differ
^
dvc/utils/table.py:24:5: note: This violates the Liskov substitution principle
dvc/utils/table.py:24:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
dvc/utils/table.py:24:5: error: Argument 8 of "add_column" is incompatible with
supertype "Table"; supertype defines the argument type as
"Union[Literal['fold'], Literal['crop'], Literal['ellipsis'], Literal['ignore']]"
[override]
def add_column( # pylint: disable=arguments-differ
^
dvc/utils/table.py:24:5: error: Argument 12 of "add_column" is incompatible
with supertype "Table"; supertype defines the argument type as "Optional[int]"
[override]
def add_column( # pylint: disable=arguments-differ
^
Found 3 errors in 1 file (checked 128 source files)
```
pylint:
```
************* Module dvc.utils.table
dvc/utils/table.py:24:4: W0237: Parameter 'vertical' has been renamed to 'overflow' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'overflow' has been renamed to 'width' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'width' has been renamed to 'min_width' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'min_width' has been renamed to 'max_width' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'max_width' has been renamed to 'ratio' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'ratio' has been renamed to 'no_wrap' in overridden 'Table.add_column' method (arguments-renamed)
dvc/utils/table.py:24:4: W0237: Parameter 'no_wrap' has been renamed to 'collapse' in overridden 'Table.add_column' method (arguments-renamed)
```
Initial simple separation to get the process of extracting dvc-objects
and dvc-data packages out of dvc.
dvc-objects is for:
* defining base Object (aka HashFile) and ObjectDB
* providing tools to work with those objects (check/transfer/etc)
* etc
dvc-data is for:
* defining higher level objects (e.g. Tree)
* bulding objects from the workspace (aka staging)
* projecting objects back into the workspace (aka checkout)
* comparing Tree objects (aka diff)
* defining misc object databases (e.g LocalObjectDB)
* higher-level transfer (e.g. with Tree indexes)
* etc
For example, the objects we use in run-cache are based on dvc-objects
and not on dvc-data, as they have absolutely nothing to do with data
management.
There are still some mixups in the code that will be addressed in the
followups.
`dvc.fs.utils.test_links()` fails with a CacheLinkError if the target `to_path` does not already exist, as is the case when "abc.dvc" exists (referring to a directory object) and we try to run "dvc pull abc/def". To avoid this issue, fully create `to_path` before attempting to create a link in it.
* test: fix test failure in case of different default branch
When the git config has set global `init.defaultBranch` different than master,
a few test cases were failing.
Currently, the test case is assuming the default branch to be set as
master.
This commit will introduce **initial_branch** option to git init call
which will ensure *master* branch is used as default for out test cases.
Set default git branch to main for test CI
Since main is going to be new default branch, it will let the test case
handle the scenario where default branch is set other than master.
Introduce isolation for git configuration
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
fix: #7196
1. Refactor exp_ref_by_name in `dvc/repo/experiments/utils.py`.
In previous the exp_ref_by_name walk throught all of the references to
match only one exp name. This makes multi exp name seeking to be a job
of `O(mn)` complexity, Walking though the exp ref and return all of the
exp ref with greatly reduce this complexity to `O(n)`
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
`@memoize()` memory lives as long as decorated function lives, which is
permanent in this case. Since `self` is referenced as a memory key no
`Index` will ever be freed.
Changed to use `@cached_property` whose cache lives as long as the
parent instance.
SCM is used for both reading and writing operations, but in the former case -
as filesystem and in the latter - as actual scm/git. So we should distinguish
those and use fs directly, which follows the overall philosophy of being able
to use Repo on any fs.
Keeping `rev=` option for `Repo` because it is a very convenient way to get
repo instance from local path but for particular branch/tag/etc.
Pre-requisite for git/dvc/repofs refactorings.
When filling missing values with `ui.rich_text` (i.e. in experiments show CMD), those values were not being correctly matched against `self._fill_value`)
The function `_raise_error_if_all_disabled` is copyed wrongly to this
place during sub command refactoring. It used to be used in `exp gc`,
can be deleted from this function.
These options are part of the default config for each particular remote.
Plugins will define their default config on theirown, but for now we keep
those in one config schema.
Hoping to fix `dvc[s3]` installations in pipenv. Pipenv's resolver is in loop, backtracking ruamel.yaml from 0.17.17 to 0.17.11 continuously. I think this has to do
more with `s3`'s dependency than `ruamel.yaml` as it only fails here. And, we are specifying dependency in 1.0.1 whereas aiobotocore has released 2.0 versions which is what is being installed, so it's redundant anyway. I am hopeful that this may help resolver with the simplification in constraints.
Adds new argument `fill_value` to ParallelCoordinatesRenderer.
For scalar columns, prevent converting to categorical by explictly inserting None when value is missing (value == fill_value).
For categorical columns, replace `fill_value` with `"Missing"` and ensure it is the last item so it gets rendered on top of the column.
- partially refactor existing pidfile behavior into using proc manager
- old pidfiles are now json-serialized executor info files placed in
proc manager subdirs (alongside proc manager pidfile/logs/etc)
- move multiprocessing based (attached) execution into separate
execution path from future "detached" execution support
- introduce ExecutorManager classes
- move executor init + execution from experiments.__init__ into the
manager classes
- make executors serializable via ExecutorInfo
related to #5489
Current dvc/command/experiments.py is too long for switch between cmd class
and the argument parser.
Split the sub command in dvc/command/experiments.py to individual files
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
We didn't mock `os.kill` in `test_dead_process` and the pid it
kills might coincidentlly be the same with some real process.
1. Add a mocker to mock out `os.kill` in `test_dead_process`.
2. Use reset_mock to replace setting it again.
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.
This PR also removes some redundant/broken remote tests.
fix: #7062
1. Add `send_signal`, `kill`, `terminate` to processManager.
2. Add tests for each of them.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
This is a very early version of test utils for dvc plugins. This module
consists of existing tests with minimal changes and is going to evolve
a lot as our plugins are getting filled in and pytest-test-utils matures.
This commit removes dvc specific message hardcoded in
scm.detach_head. Now that we are decoupling scm, we are
adding `client=` kwarg to be set by the caller.
Also adds a wrapper for DVC to use as GitFileSystem in dvc.fs.scm.
This is temporary, and will be renamed back to dvc.fs.git after the
split.
Note that this filesystem is not fully compliant, it's only
meant to be compatible with our previous implementation.
This removes dependency from dvc.utils and dvc.types from DVC.
Some utils like fix_env, is_binary have been reimplemented.
Regarding fix_env and is_binary, I think eventually this
should be a responsibility of DVC itself to set.
relpath implementation in DVC also works with paths in
different drives in Windows. I have changed this to use
os.path.relpath for now.
This adds a support for reporting progress from Git in a structured
format. The gitpython already provides somewhat structured progress
reporting, dulwich on the other hand only provides git-like line
messages.
As gitpython uses git binary, it already parse this line format
in a structured way. This PR makes dulwich use the same machinery
to unify progress reporting for both dulwich and gitpython python.
This does add a dependency on gitpython even on dulwich backend,
so I am iffy if we should do this.
The alternative is to translate gitpython's progress reports to
string, and just use string messages as progress reports on both
dulwich and gitpython.
Sometimes, `tmp_dir` fixtures take longer than a second. Using `func_only`
makes it only consider duration of test function run, excluding fixtures
duration.
The `exp init` tests may just keep waiting on the prompts blocking tests.
The blocking of prompts is likely a failure in tests setup or a bug
in the implementation. It's hard to find out these failures or avoid it.
If they take longer than 1 second, they most likely have failed as
most of test takes 0.1 second at max.
pygit2 wheels should be now available.
Had to skip pytest-docker for Windows + Python 3.10 due to
a dependency issue with it's dependency, see:
https://github.com/docker/docker-py/issues/2902.
Because our machine name now is the key value of the machine config
dict. And the machine config value will been overwritten by it when the
config is loaded. We can remove the field `name` from the config.
Borrowed this idea from pandas api. Current dropna methods will drop a
column or row if any data is missing. But it is also quite useful to
drop those columns or rows only if all of its elements are missing. This
is mostly used for pruning a table.
- allow mismatched RSA signature algo names in initial
SSH_MSG_USERAUTH_PK_OK response
- unit test overridden asyncssh publickey auth compatibility with
github ssh server
- vendor should use ssh-agent or default IdentityFile(s) when no SSH key
has been provided by the user
- encrypted (fallback) keys will always be passed to ssh-agent
fix: #6940
`--global/system/project/local` help message are in some irrelevant
command `status`, `ssh`, `create`, `destroy`. These commands are more
related to running instances rather than `machine config`, level
options didn't work for them.
1. Remove `--global/system/project/local` options in `status`, `ssh`,
`create`, `destroy`.
2. Add a test for it.
3. Correct a typo error.
fix: #6838
1. make machine status name optional.
2. show all machine status if no name provide.
3. UI changes for this cmd.
4. add a new test for it.
Co-authored-by: Dave Berenbaum <[email protected]>
All three of those use more or less the same logic for testing different
link types and maybe then using them. So now instead of having 3
implementations, we'll have just one.
Filesystem `transfer` now will not try to catch particular errno's, but
instead will just ignore any OSError and move down to the next link type
to try out, and if the last one fails - it will raise the last exception,
so if it is a common problem - we'll be able to see it.
This patch also removes `odb.confirmed_cache_type`, as it was caching
link type that is not really a cache-level thing, but rather a checkout thing,
that should only be cached for particular objects we are linking, as multiple
objects could be linked to different filesystems from the same odb (e.g.
external local output on hdd and local data on your ssd).
Next step here will be to introduce metadata diff, so that we can get rid of
temporary Link wrapper and introduce Transfer and corresponding transfer
config, that will know what links to use and could have some other things
like callbacks, so we don't continue passing dozens of kwargs along the stack.
Continuing from #6963Fixes#6980
After #6814 the header of `diff_table` is `a_rev/`b_rev` instead of `Old`/`New` so the name of the option was counterintuitive.
In addition, make `a_rev` show by default, for consistency with output format of other `diff_table` uses (`dvc metrics diff` / `dvc params diff`).
It'll give a error message to the user and reprompt them.
This mechanism can be extended to other prompts, and we
can show a warning message and/or reprompt them based on
validations.
Closes#6865. Part of #6446.
This method mimics pandas `drop_duplicates` when axis=rows.
For axis=cols this method will drop columns with 0-variance (only
1 unique value). NaNs (`_fill_value`) won't be counted as unique values.
So the following column (`-` would be the `_fill_value`) will be dropped:
foo
-
foo
foo
Intermediate PR in removing path_info from our code base.
It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.
For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).
Fixes#6774
It comes with the support for no. of parameters that matches with json.dumps.
See: https://github.com/willmcgugan/rich/pull/1547 and
https://github.com/willmcgugan/rich/pull/1644.
Also added support for printing json to stderr. Before we used to use json.dumps
and rich.json.JSON based on isatty. Now we use rich.print_json on both
tty and non-tty conditions, which may avoid different behaviours in the
future.
Looks like azure-core is dependent on a dependency `chardet` that aiohttp brought it with, which has been replaced with `charset_normalizer`.
But azure-core depends on internal of aiohttp and looks like it tries to set it's own encoding parser, `chardet` which is now not available.
Writing at the end of each step (create_summary) can mess up with some ML Framework progress bars, I moved the message to the monitor task, as the path won't change during training.
Fixesiterative/dvclive#177
We were previously using relpath in *all* cases, mainly to support WSL but relpath doesn't work
on macOS.
Added a check for using relpath *only* for WSL.
Fixes#6092
Currently whether or not link confirmation will run depends on
particular filesystem and existing workspace link type, which introduces
inconsistency in behaviour, where we sometimes catch unsupported links
before touching user data and sometimes we don't.
Depending on a filesyste (e.g. on NTFS), `_remove` might reset
read-only permissions in order to delete a hardlink to protected object,
which will also reset it for the object itself, making it unprotected,
so we need to protect it back.
Currently it depends on the whole `info()` output, which has too much
information that doesn't really matter. To check if file contents didn't
change mtime, size, inode combo is enough.
* executor: move local specific git initialization out of base
* machine: return kwargs for executor init instead of executor object
* executor: add initial ssh executor
* ssh-executor: disable strict host checking for dvc-machine instances
* initial tests for ssh executor
* drop GIT_SSH_COMMAND usage
* push: accept destination odb as a parameter
* separate cache initialization into executor.init_cache
* add test for ssh executor cache init
* remove dead code
* machine: fixture-ize default test scenarios
* test SSHExecutor.from_machine
* tpi: return default SSH port for constructing executors
* iterate on ui, has tree and shows to-be-generated yaml content
* order fs tree structure alphabetically
* always show the heading
* hide dvclive on tree if not on live mode
* Change message at the end
* s/skip/omit
* add link on the header
* remove newline on non-interactive mode
* Add experiment name check.
1. Add experiment name check (https://git-scm.com/docs/git-check-ref-format)
2. Add duplicate exp name check.
3. Add some unit test for it.
* Ban slash / in dvc exp names
* Use dulwich backend for ref name checking
* Some bug fix
* Update dvc/repo/experiments/__init__.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/repo/experiments/__init__.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some review changes
* Make some funtion more reusable.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
If supported, reflink, will provide the fastest transfer possible.
That being said, right now we use `move` in most of the places,
which still takes presidence.
Pre-requisite for #6387
scm.install does 3 different things, and all of those are
very much tied to DVC. So, as we are splitting scm out,
it should now belong to `dvc.install`.
Linux/Windows machines have only 2 cores, whereas macOS machines have 3 cores. Maybe we are stretching available resources by 2x, let's see if `-n=auto` improves test freezes.
* git: support client kwargs for auth in remote methods
* tests: use docker for git/ssh server
* move dulwich backend into subdir
* add base sync<->async support
* dulwich: add initial asyncssh SSH vendor
* dulwich: use asyncssh as the default SSH vendor
* add test for git over ssh
* test read (fetch) over ssh
* ci: re-enable ssh tests
temporary workaround for #6788, #6789
* move async stubs into dvc.scm.asyn
* tests: use sshfs and not dvc.fs.ssh
* allow explicit 0-length read in asyncssh vendor
* `dvc machine`: add a new command `rename` to it
fix#6481
1. add new command `dvc machine rename`
2. add a unit test for the call.
3. add several functional tests for it.t push myself fix6481
* Move rename logic to tpi
* use tpi.state_mv instead of rename
* Rebase to a newer version
* Update tpi version
* Remove unused assert and try catch
* throw error if new parameters are added by dvc run exp -S
Throws an error if dvc run exp --set-param references a parameter key path
that is not in the target config file. This is done to prevent silent errors
in experimentation when new params are accidentally added instead of modifying
current ones due to typos.
This implementation is based on initial suggestions in issue #5477.
* fix mypy confusion from overwriting a var
* tests: remove modify param test that was adding params
One of the test in test_modify_params was still adding a parameter
* tests: remove param modification during exp run in test_untracked
Spurious usage of modifying a parameter when the params.yaml is not yet existing in test_untracked.
* remove benedict dependency for verifying params
Instead, the keypath util module from benedict is adapted into the dvc codebased and used
natively to verify that a keypath exists within a dict.
* exp: list all invalid param modifications in error message
* exp-run: more specific exception for new params via --set-param
Uses established MissingPramsError instead of DvcException. Modified message to be more similar to other uses of this exception.
* Revert "remove benedict dependency for verifying params"
This reverts commit 6170f16dcc4033667fbacb0d0bc13c60898f58c4.
* exp: use internal benedict import for validating params
* exp: combine new param validation into merge_params
Moves new param validation into existing merge_params function as an optional check
* exp: minor ammendments to error message
* Revert "tests: remove param modification during exp run in test_untracked"
This reverts commit c9eb1a62af58586509ded362cc79e75ddc4b119c.
* exp: fix test_untracked to not fail due to new params
Fixed by creating the params file with the new parameter already present.
* machine: Add a new command `dvc machine status`
fix#6482
1. add a new command `dvc machien status`
2. add a unit test for the cli call.
3. add a unit test for the shown result.
* Update dvc/command/machine.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Update dvc/command/machine.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Update machine.py
* Remove sensitive fields
* Update dvc/machine/__init__.py
Co-authored-by: Casper da Costa-Luis <[email protected]>
* Update tests/func/machine/__init__.py
Co-authored-by: Casper da Costa-Luis <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use white list to limit shown fields
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Casper da Costa-Luis <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* dvc: drop 3.6 support
Considering that 3.6 has ~2 months until EOL, and our deps are
dropping support for it already (e.g. pygit2), we are droppipping
it as well. This will reduce the load on our test CI as well, since
we've started supporting 3.10 recently.
* setup.cfg: adjust requirements
* daemon: remove 3.6 logic
* tests: utils: remove pre 3.7 logic
* use setuptools_scm and build_requires
* remove kwargs from setup.py
* fix tests; fetch in depth in github actions
* use build to build sdist/wheel
* try updating pip and setuptools
* update pip
* remove or unpin wheel
* bring back the version in pyinstaller
* Update scripts/build.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* advertise Python 3.10 support
Pygit2 does not have windows wheels yet and no pyarrow
for 3.10. But these platform-dependent things and
remote-dependent things should not stop us from
advertising support as we are pure-python based
package and we cannot claim support for all of our
dependencies.
* set pygit2 for Python 3.6 to <1.7
Dependabot was throwing Illformed requirements `"< '3.10'"` for the
hdfs. But I noticed that it did accept other requirements where we
do similar. So it seems the indentation fixes it.
Even then, dependabot did not update any dependencies. I removed
setup.py file, after which immediately it started creating PRs.
So, I recknoed that it must have been due to some mechanism
in place from dependabot to disallow execution, so I tried to
remove few blocks of code from setup.py. And, I was eventually
able to figure out that it was from the block where we import version.
So, I added a try-except block and it magically again worked. :)
Let's see how it goes.
1. update diff.py,experiments.py,metrics.py,params.py,status.py to
print prettify json
2. rewrite write_json in __init__.py to accept default function
3. add new unit test for write_json in tests/unit/ui/test_console.py
4. Fix#6712
Co-authored-by: Zhenyu Gao <[email protected]>
* doctor: fixes unknown cache director error on some windows mapped network shares
On Windows, resolve calls os.path._getfinalpathname, which calls GetFinalPathNameByHandle.
This starts from the final resolved path of the file in the NT namespace (e.g. \Device\Mup\server\share...)
and works backward as required to get the VOLUME_NAME_DOS. If the final path is a UNC share (i.e. \Device\Mup),
We fix this by calling resolve() also on the partition mountpoints
We also take the opportunity to make checks in case a PermissionError is returned to us
Fixes github.com/iterative/dvc/issues/6305
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* doctor: fixes unknown cache director error on some windows mapped network shares
On Windows, resolve calls os.path._getfinalpathname, which calls GetFinalPathNameByHandle.
This starts from the final resolved path of the file in the NT namespace (e.g. \Device\Mup\server\share...)
and works backward as required to get the VOLUME_NAME_DOS. If the final path is a UNC share (i.e. \Device\Mup),
We fix this by calling resolve() also on the partition mountpoints
We also take the opportunity to make checks in case a PermissionError is returned to us
Fixes#6305
Co-authored-by: CHETOUAN <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: CHETOUAN <[email protected]>
* exp init: introduce `--with={default,live}` flag
This flag will select the different mode of outputs of stage.
With `--with=live`, it'll enable `live` setting in the stage.
On interactive mode, it'll always ask user if they want to
use live (unless `--with=live` is specified), and then it'll
skip metrics/plots question and moves on to ask for path to
dvclive path.
Also, if live setting is enabled in the stage, it'll set the
`models` to be a checkpoint output.
* s/with/type and don't ask for live interactively
1. update ls __init__.py to output pretty JSON format
2. introduce write_json function into ui __init__.py
Co-authored-by: Zhenyu Gao <[email protected]>
This was used before on opening http urls as a stream,
but we no longer use those (as we use fsspec's open method).
One thing to note however is that, compared to previous
implementation, fsspec's http filesystem's open is not very
robust to failures.
speedcopy only patches shutil.copyfile, which we don't even use
anywhere because it lacks callbacks/progress and we instead use
our own copyfile implementation, so this is totally useless for
us and can be removed.
```
2021-09-21T00:15:31.6789059Z ____________________ ERROR at setup of test_get_url[azure] _____________________
2021-09-21T00:15:31.6790443Z [gw0] linux -- Python 3.7.12 /opt/hostedtoolcache/Python/3.7.12/x64/bin/python
2021-09-21T00:15:31.6791202Z
2021-09-21T00:15:31.6792291Z request = <FixtureRequest for <Function test_get_url[azure]>>
2021-09-21T00:15:31.6793561Z
2021-09-21T00:15:31.6794120Z def fill(request):
2021-09-21T00:15:31.6794792Z item = request._pyfuncitem
2021-09-21T00:15:31.6795627Z fixturenames = getattr(item, "fixturenames", None)
2021-09-21T00:15:31.6797090Z if fixturenames is None:
2021-09-21T00:15:31.6797922Z fixturenames = request.fixturenames
2021-09-21T00:15:31.6798497Z
2021-09-21T00:15:31.6799393Z if hasattr(item, 'callspec'):
2021-09-21T00:15:31.6800256Z for param, val in sorted_by_dependency(item.callspec.params, fixturenames):
2021-09-21T00:15:31.6801309Z if val is not None and is_lazy_fixture(val):
2021-09-21T00:15:31.6802233Z > item.callspec.params[param] = request.getfixturevalue(val.name)
2021-09-21T00:15:31.6802912Z
2021-09-21T00:15:31.6803937Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pytest_lazyfixture.py:35:
2021-09-21T00:15:31.6804952Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-09-21T00:15:31.6806085Z /home/runner/work/dvc/dvc/tests/remotes/azure.py:102: in azure
2021-09-21T00:15:31.6807090Z container.get_container_properties()
2021-09-21T00:15:31.6808748Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/core/tracing/decorator.py:83: in wrapper_use_tracer
2021-09-21T00:15:31.6822551Z return func(*args, **kwargs)
2021-09-21T00:15:31.6824143Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/storage/blob/_container_client.py:497: in get_container_properties
2021-09-21T00:15:31.6825038Z process_storage_error(error)
2021-09-21T00:15:31.6825652Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-09-21T00:15:31.6825908Z
2021-09-21T00:15:31.6826916Z storage_error = HttpResponseError("Operation returned an invalid status 'The value for one of the HTTP headers is not in the correct format.'")
2021-09-21T00:15:31.6827586Z
2021-09-21T00:15:31.6828351Z def process_storage_error(storage_error): # pylint:disable=too-many-statements
2021-09-21T00:15:31.6829086Z raise_error = HttpResponseError
2021-09-21T00:15:31.6829597Z serialized = False
2021-09-21T00:15:31.6830066Z if not storage_error.response:
2021-09-21T00:15:31.6830554Z raise storage_error
2021-09-21T00:15:31.6831558Z # If it is one of those three then it has been serialized prior by the generated layer.
2021-09-21T00:15:31.6832480Z if isinstance(storage_error, (PartialBatchErrorException,
2021-09-21T00:15:31.6833636Z ClientAuthenticationError, ResourceNotFoundError, ResourceExistsError)):
2021-09-21T00:15:31.6834687Z serialized = True
2021-09-21T00:15:31.6835753Z error_code = storage_error.response.headers.get('x-ms-error-code')
2021-09-21T00:15:31.6836524Z error_message = storage_error.message
2021-09-21T00:15:31.6837001Z additional_data = {}
2021-09-21T00:15:31.6837398Z error_dict = {}
2021-09-21T00:15:31.6837739Z try:
2021-09-21T00:15:31.6838506Z error_body = ContentDecodePolicy.deserialize_from_http_generics(storage_error.response)
2021-09-21T00:15:31.6839345Z # If it is an XML response
2021-09-21T00:15:31.6839837Z if isinstance(error_body, Element):
2021-09-21T00:15:31.6840285Z error_dict = {
2021-09-21T00:15:31.6840750Z child.tag.lower(): child.text
2021-09-21T00:15:31.6841435Z for child in error_body
2021-09-21T00:15:31.6841822Z }
2021-09-21T00:15:31.6842195Z # If it is a JSON response
2021-09-21T00:15:31.6842692Z elif isinstance(error_body, dict):
2021-09-21T00:15:31.6843413Z error_dict = error_body.get('error', {})
2021-09-21T00:15:31.6843893Z elif not error_code:
2021-09-21T00:15:31.6844347Z _LOGGER.warning(
2021-09-21T00:15:31.6845611Z 'Unexpected return type % from ContentDecodePolicy.deserialize_from_http_generics.', type(error_body))
2021-09-21T00:15:31.6846887Z error_dict = {'message': str(error_body)}
2021-09-21T00:15:31.6847294Z
2021-09-21T00:15:31.6847730Z # If we extracted from a Json or XML response
2021-09-21T00:15:31.6848205Z if error_dict:
2021-09-21T00:15:31.6848792Z error_code = error_dict.get('code')
2021-09-21T00:15:31.6849704Z error_message = error_dict.get('message')
2021-09-21T00:15:31.6850578Z additional_data = {k: v for k, v in error_dict.items() if k not in {'code', 'message'}}
2021-09-21T00:15:31.6851392Z except DecodeError:
2021-09-21T00:15:31.6851801Z pass
2021-09-21T00:15:31.6852290Z
2021-09-21T00:15:31.6852601Z try:
2021-09-21T00:15:31.6853156Z # This check would be unnecessary if we have already serialized the error
2021-09-21T00:15:31.6853821Z if error_code and not serialized:
2021-09-21T00:15:31.6854423Z error_code = StorageErrorCode(error_code)
2021-09-21T00:15:31.6855164Z if error_code in [StorageErrorCode.condition_not_met,
2021-09-21T00:15:31.6856229Z StorageErrorCode.blob_overwritten]:
2021-09-21T00:15:31.6856938Z raise_error = ResourceModifiedError
2021-09-21T00:15:31.6857739Z if error_code in [StorageErrorCode.invalid_authentication_info,
2021-09-21T00:15:31.6858641Z StorageErrorCode.authentication_failed]:
2021-09-21T00:15:31.6859468Z raise_error = ClientAuthenticationError
2021-09-21T00:15:31.6860256Z if error_code in [StorageErrorCode.resource_not_found,
2021-09-21T00:15:31.6861256Z StorageErrorCode.cannot_verify_copy_source,
2021-09-21T00:15:31.6861998Z StorageErrorCode.blob_not_found,
2021-09-21T00:15:31.6862716Z StorageErrorCode.queue_not_found,
2021-09-21T00:15:31.6863718Z StorageErrorCode.container_not_found,
2021-09-21T00:15:31.6864526Z StorageErrorCode.parent_not_found,
2021-09-21T00:15:31.6865432Z StorageErrorCode.share_not_found]:
2021-09-21T00:15:31.6866549Z raise_error = ResourceNotFoundError
2021-09-21T00:15:31.6867609Z if error_code in [StorageErrorCode.account_already_exists,
2021-09-21T00:15:31.6868499Z StorageErrorCode.account_being_created,
2021-09-21T00:15:31.6869383Z StorageErrorCode.resource_already_exists,
2021-09-21T00:15:31.6870728Z StorageErrorCode.resource_type_mismatch,
2021-09-21T00:15:31.6871688Z StorageErrorCode.blob_already_exists,
2021-09-21T00:15:31.6872429Z StorageErrorCode.queue_already_exists,
2021-09-21T00:15:31.6873252Z StorageErrorCode.container_already_exists,
2021-09-21T00:15:31.6874124Z StorageErrorCode.container_being_deleted,
2021-09-21T00:15:31.6874922Z StorageErrorCode.queue_being_deleted,
2021-09-21T00:15:31.6875857Z StorageErrorCode.share_already_exists,
2021-09-21T00:15:31.6876608Z StorageErrorCode.share_being_deleted]:
2021-09-21T00:15:31.6877289Z raise_error = ResourceExistsError
2021-09-21T00:15:31.6877807Z except ValueError:
2021-09-21T00:15:31.6878259Z # Got an unknown error code
2021-09-21T00:15:31.6878648Z pass
2021-09-21T00:15:31.6878942Z
2021-09-21T00:15:31.6879424Z # Error message should include all the error properties
2021-09-21T00:15:31.6879908Z try:
2021-09-21T00:15:31.6880416Z error_message += "\nErrorCode:{}".format(error_code.value)
2021-09-21T00:15:31.6881289Z except AttributeError:
2021-09-21T00:15:31.6883082Z error_message += "\nErrorCode:{}".format(error_code)
2021-09-21T00:15:31.6883807Z for name, info in additional_data.items():
2021-09-21T00:15:31.6884446Z error_message += "\n{}:{}".format(name, info)
2021-09-21T00:15:31.6884932Z
2021-09-21T00:15:31.6885741Z # No need to create an instance if it has already been serialized by the generated layer
2021-09-21T00:15:31.6886538Z if serialized:
2021-09-21T00:15:31.6887019Z storage_error.message = error_message
2021-09-21T00:15:31.6887526Z error = storage_error
2021-09-21T00:15:31.6887908Z else:
2021-09-21T00:15:31.6888499Z error = raise_error(message=error_message, response=storage_error.response)
2021-09-21T00:15:31.6889380Z # Ensure these properties are stored in the error instance as well (not just the error message)
2021-09-21T00:15:31.6890078Z error.error_code = error_code
2021-09-21T00:15:31.6890630Z error.additional_info = additional_data
2021-09-21T00:15:31.6891956Z # error.args is what's surfaced on the traceback - show error message in all cases
2021-09-21T00:15:31.6892672Z error.args = (error.message,)
2021-09-21T00:15:31.6893112Z try:
2021-09-21T00:15:31.6893784Z # `from None` prevents us from double printing the exception (suppresses generated layer error context)
2021-09-21T00:15:31.6894873Z > exec("raise error from None") # pylint: disable=exec-used # nosec
2021-09-21T00:15:31.6896126Z E azure.core.exceptions.HttpResponseError: The value for one of the HTTP headers is not in the correct format.
2021-09-21T00:15:31.6897385Z E RequestId:d83b8102-bde1-4a77-aa7a-b67bbc2c4a80
2021-09-21T00:15:31.6898080Z E Time:2021-09-21T00:09:53.815Z
2021-09-21T00:15:31.6898578Z E ErrorCode:InvalidHeaderValue
2021-09-21T00:15:31.6899259Z E headername:x-ms-version
2021-09-21T00:15:31.6900066Z E headervalue:2020-10-02
2021-09-21T00:15:31.6900355Z
2021-09-21T00:15:31.6901781Z /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/storage/blob/_shared/response_handlers.py:177: HttpResponseError
```
* rebased PR 6111 onto master
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use make_tmp_dir fixture in tests
Co-authored-by: U119511 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Alexander Meier <[email protected]>
Co-authored-by: meierale <[email protected]>
* tests: additional callbacks test for remaining filesystems
* requirements: bump pydrive2 to 1.9.4
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Trying to see if getting rid of this dependency, our issue with timeout
gets solved or not.
We use timeout-minutes anyway on GHA, so we probably don't need it.
Locally, users can monitor this themselves.
* machine: add `machine list`&`machine modify`
1. add new command `dvc machine list`
2. complete `dvc machine modify`
3. add new unit tests for this two commands to ensure the call.
4. add functionality tests for two commands
* Update dvc/config_schema.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update tests/func/machine/test_machine_config.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some reviewed problems
1. capitalize global variables.
2. remove precision restriction.
3. solve windows path seperator problem.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* dvc cli: fixed error in 'dvc config --list' in non repo dir
Fixes#6496
* Issue 6496: Added test that covers the issue
* dvc cli: fixes error in 'dvc config <name>' in non dvc repo dir
Fixes the specified error. Did some refactoring based on previous commits.
Also added a test case for setting config in non dvc repo dir.
Fixes#6496
* Issue-6496: dvc config --<level> and dvc config --list --<level> will now throw config error if ran in non dvc repo with repo only level
* Issue 6496: Segregated config levels into repo and system levels that helped in simplifying the logic to decide appropriate level at the time of cli operation.
The current ci failure was caused by wrong order of the experiments.
Because our test script runs so fast that they might share the same
commit timestamp, that makes them in an uncertain order. And then gives
a wrong branch_base.
* output: use remote kwarg instead of odb
Fixes#6596
* Update dvc/output.py
Co-authored-by: Dave Berenbaum <[email protected]>
* add test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Dave Berenbaum <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fs.download: add callback support
* fix lint issues
* compat for gdrive, use fs to upload text for tests
* fix typing issues
* skip gdrive test
* Update tests/func/test_fs.py
* fs.upload: add support for optional callback
Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
<https://dvc.org/doc/get-started>`_ docs to better understand what DVC
does and how it can fit your scenarios.
The closest *analogies* to describe the main DVC features are these:
#.**Git for data**: Store and share data artifacts (like Git-LFS but without a server) and models, connecting them with a Git repository. Data management meets GitOps!
#.**Makefiles** for ML: Describes how data or model artifacts are built from other data and code in a standard format. Now you can version your data pipelines with Git.
#. Local **experiment tracking**: Turn your machine into an ML experiment management platform, and collaborate with others using existing Git hosting (Github, Gitlab, etc.).
Git is employed as usual to store and version code (including DVC meta-files as placeholders for data).
DVC `stores data and model files <https://dvc.org/doc/start/data-management>`_ seamlessly in a cache outside of Git, while preserving almost the same user experience as if they were in the repo.
To share and back up the *data cache*, DVC supports multiple remote storage platforms - any cloud (S3, Azure, Google Cloud, etc.) or on-premise network storage (via SSH, for example).
|Flowchart|
`DVC pipelines <https://dvc.org/doc/start/data-management/pipelines>`_ (computational graphs) connect code and data together.
They specify all steps required to produce a model: input dependencies including code, data, commands to run; and output information to be saved.
Last but not least, `DVC Experiment Versioning <https://dvc.org/doc/start/experiments>`_ lets you prepare and run a large number of experiments.
Their results can be filtered and compared based on hyperparameters and metrics, and visualized with multiple plots.
.._`VS Code Extension`:
Visual Studio Code Extension
============================
|VS Code|
To use DVC as a GUI right from your VS Code IDE, install the `DVC Extension <https://marketplace.visualstudio.com/items?itemName=Iterative.dvc>`_ from the Marketplace.
It currently features experiment tracking and data management, and more features (data pipeline support, etc.) are coming soon!
|VS Code Extension Overview|
Note: You'll have to install core DVC on your system separately (as detailed
below). The Extension will guide you if needed.
Installation
============
There are four options to install DVC: ``pip``, Homebrew, Conda (Anaconda) or an OS-specific package.
There are several ways to install DVC: in VS Code; using``snap``, ``choco``, ``brew``, ``conda``, ``pip``; or with an OS-specific package.
Full instructions are `available here <https://dvc.org/doc/get-started/install>`_.
Snap (Snapcraft/Linux)
----------------------
Snapcraft (Linux)
-----------------
|Snap|
@@ -102,11 +119,10 @@ Snap (Snapcraft/Linux)
snap install dvc --classic
This corresponds to the latest tagged release.
Add ``--beta`` for the latest tagged release candidate,
or ``--edge`` for the latest ``master`` version.
Add ``--beta`` for the latest tagged release candidate, or ``--edge`` for the latest ``main`` version.
Choco (Chocolatey/Windows)
--------------------------
Chocolatey (Windows)
--------------------
|Choco|
@@ -114,8 +130,8 @@ Choco (Chocolatey/Windows)
choco install dvc
Brew (Homebrew/Mac OS)
----------------------
Brew (mac OS)
-------------
|Brew|
@@ -123,8 +139,8 @@ Brew (Homebrew/Mac OS)
brew install dvc
Conda (Anaconda)
----------------
Anaconda (Any platform)
-----------------------
|Conda|
@@ -133,11 +149,10 @@ Conda (Anaconda)
conda install -c conda-forge mamba # installs much faster than conda
mamba install -c conda-forge dvc
Depending on the remote storage type you plan to use to keep and share your data, you might need to
Depending on the remote storage type you plan to use to keep and share your data, you might need to install optional dependencies: `dvc-s3`, `dvc-azure`, `dvc-gdrive`, `dvc-gs`, `dvc-oss`, `dvc-ssh`.
pip (PyPI)
----------
PyPI (Python)
-------------
|PyPI|
@@ -145,10 +160,8 @@ pip (PyPI)
pip install dvc
Depending on the remote storage type you plan to use to keep and share your data, you might need to specify
one of the optional dependencies: ``s3``, ``gs``, ``azure``, ``oss``, ``ssh``. Or``all``to include them all.
The command should look like this: ``pip install dvc[s3]`` (in this case AWS S3 dependencies such as ``boto3``
will be installed automatically).
Depending on the remote storage type you plan to use to keep and share your data, you might need to specify one of the optional dependencies: ``s3``, ``gs``, ``azure``, ``oss``, ``ssh``. Or ``all`` to include them all.
The command should look like this: ``pip install 'dvc[s3]'`` (in this case AWS S3 dependencies such as``boto3``will be installed automatically).
To install the development version, run:
@@ -156,88 +169,58 @@ To install the development version, run:
pip install git+git://github.com/iterative/dvc
Package
-------
Package (Platform-specific)
---------------------------
|Packages|
Self-contained packages for Linux, Windows, and Mac are available. The latest version of the packages
can be found on the GitHub `releases page <https://github.com/iterative/dvc/releases>`_.
Self-contained packages for Linux, Windows, and Mac are available.
The latest version of the packages can be found on the GitHub `releases page <https://github.com/iterative/dvc/releases>`_.
#. Data Engineering tools such as `AirFlow <https://airflow.apache.org/>`,
`Luigi <https://github.com/spotify/luigi>`, and others - in DVC data,
model and ML pipelines represent a single ML project focused on data
scientists' experience. Data engineering tools orchestrate multiple data
projects and focus on efficient execution. A DVC project can be used from
existing data pipelines as a single execution step.
#.`Git-annex <https://git-annex.branchable.com/>`_ - DVC uses the idea of storing the content of large files (which should
not be in a Git repository) in a local key-value store, and uses file hardlinks/symlinks instead of
copying/duplicating files.
#.`Git-LFS <https://git-lfs.github.com/>`_ - DVC is compatible with many
remote storage services (S3, Google Cloud, Azure, SSH, etc). DVC also
uses reflinks or hardlinks to avoid copy operations on checkouts; thus
handling large data files much more efficiently.
#. Makefile (and analogues including ad-hoc scripts) - DVC tracks
dependencies (in a directed acyclic graph).
#.`Workflow Management Systems <https://en.wikipedia.org/wiki/Workflow_management_system>`_ - DVC is a workflow
management system designed specifically to manage machine learning experiments. DVC is built on top of Git.
#.`DAGsHub <https://dagshub.com/>`_ - online service to host DVC
projects. It provides a useful UI around DVC repositories and integrates
other tools.
#.`DVC Studio <https://studio.iterative.ai/>`_ - official online
platform for DVC projects. It can be used to manage data and models, run
and track experiments, and visualize and share results. Also, it
integrates with `CML (CI/CD for ML) <https://cml.dev/>` for training
models in the cloud or Kubernetes.
Contributing
============
|Maintainability| |Donate|
|Maintainability|
Contributions are welcome! Please see our `Contributing Guide <https://dvc.org/doc/user-guide/contributing/core>`_ for more
details. Thanks to all our contributors!
Contributions are welcome!
Please see our `Contributing Guide <https://dvc.org/doc/user-guide/contributing/core>`_ for more details.
Thanks to all our contributors!
|Contribs|
Mailing List
============
Community and Support
=====================
Want to stay up to date? Want to help improve DVC by participating in our occasional polls? Subscribe to our `mailing list <https://sweedom.us10.list-manage.com/subscribe/post?u=a08bf93caae4063c4e6a351f6&id=24c0ecc49a>`_. No spam, really low traffic.
@@ -133,3 +162,13 @@ class ParamsDependency(Dependency):
)
returnHashInfo(self.PARAM_PARAMS,info)
defsave(self):
ifnotself.exists:
raiseself.DoesNotExistError(self)
ifnotself.isfileandnotself.isdir:
raiseself.IsNotFileOrDirError(self)
self.ignore()
self.hash_info=self.get_hash()
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Paweł Redzyński
Peter Rowlands
skshetry
Gao
dberenbaum
Ivan Shcheklein
pre-commit-ci[bot]
Jorge Orpinel
David de la Iglesia Castro
Ruslan Kuprieiev
dependabot[bot]
Jorge Orpinel Perez
Alexander Guschin
Alex Hall
Amey Kulkarni
Yury
Daniele Trifirò
huang06
Alexander Schepanovski
Karthikeyan SingaravelanandDavid de la Iglesia Castro
Samuel Knight
joncrall
Nick SorrosandPaweł Redzyński
RCdeWit
Rob de WitandLaurens Duijvesteijn
Fabian ZillsandSaugat Pachhai
Casper da Costa-Luis
Lucas Cabral
Zhenyu Gao
Nikita Kodenko
tapadipti
Daniele Trifirò
Jeff Hale
Matt Seddon
Gavin Li
Jon Burdo
Amrit GhimireandSaugat Pachhai
Kwon-Young ChoiandPeter Rowlands
Gustavo José de Sousa
Dimitar Todorov
Димитър Тодоров (Dimitar Todorov)andSaugat Pachhai
Batuhan Taskaya
Guðmundur HeimissonandRuslan Kuprieiev
Ruslan Kuprieiev
Matthew Beckers
eggqq007andZhenyu Gao
Michael Lingelbach
Guillaume COTER
pwoolvett
Amey Kulkarni
whysage
Amey Kulkarni