1119 Commits
Author SHA1 Message Date
Yury 6edca79b79 repro dry: show information if the stage is cached
Fixes: #8342
2022-11-24 21:49:52 +02:00
b02855fb93 ui: Fix WSL check in open_browser (#8604)
* ui: Fix WSL check in `open_browser`

* Update dvc/ui/__init__.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-11-23 09:57:57 +00:00
karajan1001 9b9c04e633 Solve the problem during executor running
fix #7693

The problem was when we gathered experiments, and at the same time,
the executor are creating new commits. It will make the checkpoint
top/head return by `get_running_exps`/`get_ref`/`branch_revs` not the
same one. And cause the problem described in #7693

To solve this we did
1. move the `get_rev` to the beginning of `exp show` to make sure the
   return results in `branch_revs`
2. Add a new function to move all of the properties that only belong to
   `head` of the checkpoint from the wrong place to the top.
3. Add a new unit test for it.
4. Force fetch_running_exp to return data.
5. fix a bug in tmp_dir_executor.
2022-11-22 17:44:55 +08:00
karajan1001 350e0fb1f7 queue kill: we can manually mark problematic tasks as failure
fix: #8461
fix: #8473
For now, for those inactive but uncompleted tasks, they still shown as
Running in `exp show` and `task queue`, we cannot remove them because
they are regarded as running, we cann't kill them because their are no
real process behind them.

1. Now we can examine these false running tasks and mark them as failure
   in `queue kill` command. #8461
2. Better msg handle for initliaizing tasks in `queue kill`#8473
3. Add new unit tests for both cases.
4. bump into new dvc-task 0.1.6
2022-11-22 13:33:23 +08:00
dberenbaum f14358bc72 plots: add test for default x label 2022-11-21 21:05:12 +01:00
David de la Iglesia Castro 705bc6ec82 Added test_infer_x_label 2022-11-21 21:05:12 +01:00
Paweł Redzyński ba2e8bdac7 plots: data conversion: adjust for viewer backend (#8421)
* plots: vega: converter: support multi series data split and props inference

Fixes: #7754, #8399
Related: iterative/studio#4369

* plots: converter: make converter infer x and y labels if not provided

* plots: vega: converter: roll back to datapoints format

* plots: vega: converter: refactor

* plots: infer labels on

* plots: get_plot_id: fix naming

* plots: vega: converter: notes regarding converting
2022-11-18 12:36:00 +00:00
karajan1001 9fbbfd07a6 Fix some celery queue realted ci failure
fix: #8403
1. remove some of the flaky mark
2. In `get_result` make sure the celery task is completed.
2022-11-11 07:04:43 +05:45
karajan1001 929de7c651 Solve the crash on getting name of applied experiment branch
fix: #8526
1. The current `get_names` will crash if an experiment branch had
   already been given a new branch name to it.

1. Solve the crash when we getting the name of a newly fixed experiment
   branch.
2. Add a new unit test to it.
2022-11-10 13:18:45 +08:00
Saugat Pachhai (सौगात) a48a19ce31 tests: ignore versions for subprojects 2022-11-07 12:45:32 +01:00
David de la Iglesia Castro 1de983f0de Remove live section support. 2022-11-07 12:45:32 +01:00
Saugat Pachhai (सौगात) 4d17c171f1 tests: use mocker 2022-11-03 23:07:28 +05:45
Saugat Pachhai (सौगात) c02bd7f318 migrate some files to pytest 2022-11-03 23:07:28 +05:45
Saugat Pachhai (सौगात) 1cad8ea7fd migrate test_{add,analytics,init,unprotect,local,process}.py to pytest 2022-11-03 23:07:28 +05:45
skshetry 4ca2158b9d tests: retry celery queue on timeout (#8488)
retry celery queue on timeout
2022-10-31 19:38:05 +05:45
Peter Rowlands 3478a7e6a9 import-url: support cloud versioning 2022-10-19 17:12:16 +03:00
skshetry bbaf57c5b2 _debug: minor refactor on debugtools (#8454)
* _debug: minor refactor on debugtools

* delete test
2022-10-19 18:01:20 +05:45
David de la Iglesia Castro 61f8478885 tests: Remove test_ssh
Moved to dvc-ssh repo in https://github.com/iterative/dvc-ssh/pull/13
2022-10-19 12:35:11 +03:00
David de la Iglesia Castro 3bbf8fa7d2 config: ssh: Add passphrase and ask_passphrase options.
Bump `dvc-ssh`.

Fixes #8137
2022-10-14 12:11:31 +03:00
karajan1001 884182f6f2 Add "status" to replace "running" and "queued" in output of exp show
fix: #7986
1. Add two new flags  `--hide-queued` and  `--hide-failed` to `exp show`
2. Allow `exp show` to show failed experiments.
3. Add unit test for the failed experiments shown.
4. Add name support for failed exp
5. Add error msg to the `exp show` output
2022-10-06 17:20:03 +08:00
Ronan Lamy 22655fa46d ls-url: update command test for --recursive 2022-10-04 14:37:29 +03:00
Ronan Lamy fa7619dfee ls-url: create basic version of dvc ls-url command
For now, this only takes a URL (or local path) and lists its contents
if the protocol supports it. The command fails if the target doesn't
exist and returns the (absolute) path to the target if it's not a
directory.
2022-10-04 14:37:29 +03:00
Peter Rowlands 4904286307 config: wrap UnicodeDecodeErrors on load 2022-09-30 11:45:43 +09:00
Ronan Lamy 04af944e22 fs: test and slightly refactor get_fs_config() 2022-09-28 22:06:40 +03:00
Paweł Redzyński 25ea5f35ec plots: pass all types of plots in --json option
Related: iterative/vscode-dvc#1757
2022-09-27 14:23:42 +02:00
karajan1001 6bd14b21ef Move infofile operation to executor and
Currently infofile dump was spread in both executor and queue classes.
put them all into executor to make them more managable. Move all ending
dump operations to the cleanup function, because the collect_result progress
might failed because of git operations, in which can guarantee them to
be run in a `finally` scope.
2022-09-26 16:25:47 +08:00
karajan1001 c2829f4ce8 Make executor info compatible with old versions. 2022-09-26 16:25:47 +08:00
karajan1001 7fb9dcec12 Add tests for executor status
1. Move some basic test script from function tests to unit test.
2. Add success/failed tests for the status change of `tempdir`, `celery`, `workspace` running case.
2022-09-26 16:25:47 +08:00
karajan1001 90109bccd7 Refactor seperate the initialization of executor and setup environment
Reason:
1. Need some more granular control over initialization progress of
   executor.

What done:
1. Seperate `init_git` and `init_cache` progress out from setup_executor
2. Move `set_ref` into `init_git`
2022-09-26 16:25:47 +08:00
Paweł Redzyńskiandpre-commit-ci[bot] c617769a1d deps: bump codespell (#8199)
[pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/codespell-project/codespell: v2.1.0 → v2.2.1](https://github.com/codespell-project/codespell/compare/v2.1.0...v2.2.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-09-22 19:31:24 +05:45
Ruslan Kuprieiev 35e6f082a4 analytics: use iterative-telemetry for user_id lookup 2022-09-17 00:45:45 +03:00
skshetry 391bb29be2 dvcfs: rename DvcFileSystem to DVCFileSystem (#8307) 2022-09-16 18:18:55 +05:45
skshetry 8e5f4a8f35 data ls: new command to show metadata with outputs (#8252)
data ls: add new command to show metadata for the outputs
2022-09-09 17:36:24 +05:45
Saugat Pachhai (सौगात) c73764156e metadata: support custom metadata in .dvc files
This adds support for custom metadata in outputs under `meta` key.
Also adds `--meta` flag to `add`/`import`/`import-url` that supports
`key=value` values.
2022-09-08 14:10:35 +03:00
skshetry df60df5242 add metadata fields: label, type to data (#8232)
gto like metadata fields: labels and type
2022-09-08 16:33:47 +05:45
Ruslan Kuprieiev a124d7e866 deps: bump dvc-data to 0.6.3 2022-09-07 02:23:29 +03:00
Ruslan Kuprieiev 628be2ae49 deps: bump dvc-data to 0.6.1 2022-09-06 20:30:20 +03:00
Peter Rowlands 660c17f654 output: save version_id meta 2022-09-01 19:36:53 +03:00
skshetry 49da2eef02 tests: set celery ping_task_timeout 3x the default (#8221)
tests: set celery ping_timeout 3x the default
2022-09-01 16:48:19 +05:45
skshetry a8415b01d0 data status: fix quoting on command hints for untracked files (#8211)
data status: fix quoting on command hints
2022-08-30 22:51:39 +05:45
Dave Berenbaum 1d3102f29b data status: update hints to include fetch and checkout (#8209) 2022-08-30 22:05:32 +05:45
David de la Iglesia Castro ab780879d1 info: Include subprojects. 2022-08-30 09:22:39 +02:00
skshetry bfad82a2e3 data status: remove --withdirs, show unknowns in CLI (#8189)
* data status: remove --withdirs, show unknowns in CLI

* data status: only consider unknown from newer object during diff
2022-08-30 12:11:02 +05:45
skshetry e5eec59cb8 test on 3.11 (#8196)
We don't have aiohttp support yet, due to which we have to install
without cythonized extension.

Similarly, there has been some changes in how dataclasses disallows some
mutable arguments to be set as a class property, due to which
`hydra.conf` fails to import in 3.11.

See https://github.com/python/cpython/pull/29867. For now, I have
patched the module and avoid importing as a pytest-plugin.

dvc-hdfs/pyarrow is not there yet, we'll usually see a release 2-3
months after official reelase.

dvc-hdfs is skipped in 3.11 in extras_requires. `--set-params` will
return a nice error message if hydra cannot be imported in 3.11. I
decided to only patch that in tests.
2022-08-29 19:58:17 +05:45
David de la Iglesia Castro 9e2ebe9fcc resolve_rev: raise DvcException on empty repo. 2022-08-29 01:50:14 +03:00
b7aea414e6 plots templates: change ui to not dump to file
Update dvc/commands/plots.py

Co-authored-by: David de la Iglesia Castro <[email protected]>

Update tests/unit/command/test_plots.py

Co-authored-by: David de la Iglesia Castro <[email protected]>

Update dvc/commands/plots.py

Co-authored-by: Paweł Redzyński <[email protected]>

fixed test
2022-08-29 01:46:47 +03:00
skshetry 063eb6904d data status: test cleanup and improvements (#8185) 2022-08-25 18:50:46 +05:45
skshetry 2c556f58d3 data status: fix not-in-cache hint (#8181)
See https://github.com/iterative/dvc.org/pull/3812\#discussion_r936625617.
2022-08-25 15:39:59 +05:45
Daniele Trifirò 26008f1559 import-url/update: add --no-download flag
add --no-download flag to dvc import-url/dvc update to only create/update .dvc
files without downloading the associated data.

Created .dvc files can be fetched using `dvc pull` or can be
updated using `dvc update --no-download`.
2022-08-23 23:07:01 +03:00
Daniele Trifirò 92d680ff94 tests: fix typo in test name 2022-08-23 23:07:01 +03:00