Commit Graph
2759 Commits
Author SHA1 Message Date
skshetry bfad82a2e3 data status: remove --withdirs, show unknowns in CLI (#8189)
* data status: remove --withdirs, show unknowns in CLI

* data status: only consider unknown from newer object during diff
2022-08-30 12:11:02 +05:45
skshetry e5eec59cb8 test on 3.11 (#8196)
We don't have aiohttp support yet, due to which we have to install
without cythonized extension.

Similarly, there has been some changes in how dataclasses disallows some
mutable arguments to be set as a class property, due to which
`hydra.conf` fails to import in 3.11.

See https://github.com/python/cpython/pull/29867. For now, I have
patched the module and avoid importing as a pytest-plugin.

dvc-hdfs/pyarrow is not there yet, we'll usually see a release 2-3
months after official reelase.

dvc-hdfs is skipped in 3.11 in extras_requires. `--set-params` will
return a nice error message if hydra cannot be imported in 3.11. I
decided to only patch that in tests.
2022-08-29 19:58:17 +05:45
David de la Iglesia Castro 9e2ebe9fcc resolve_rev: raise DvcException on empty repo. 2022-08-29 01:50:14 +03:00
David de la Iglesia Castro 2cef130191 repo: Handle no commits for exp show and plots diff.
Make it consistent with all the other commands that use `brancher` (i.e. `dvc diff`, `metrics diff`, `params diff`)

Closes #8163
2022-08-29 01:50:14 +03:00
David de la Iglesia Castro 17b9b1571e api: Add details forparams_show stages syntax.
Explain `stages` syntax for cases where the api is used from a directory different to the one the `dvc.yam` is located.
2022-08-29 01:49:38 +03:00
b7aea414e6 plots templates: change ui to not dump to file
Update dvc/commands/plots.py

Co-authored-by: David de la Iglesia Castro <[email protected]>

Update tests/unit/command/test_plots.py

Co-authored-by: David de la Iglesia Castro <[email protected]>

Update dvc/commands/plots.py

Co-authored-by: Paweł Redzyński <[email protected]>

fixed test
2022-08-29 01:46:47 +03:00
Alex Hall 8bdf9b1138 Better error message when using remove on file instead of dvc file 2022-08-29 01:43:43 +03:00
David de la Iglesia Castro 7b3b4b373b api: Fix params_show for untracked targets.
If an untracked target is provided alongside `stages` or `deps`, don't load the parameters from the target and raise no params found instead.

Closes #8161
2022-08-26 12:00:45 +02:00
Peter Rowlands (변기호) 512bfc4305 tests: mark exp queue func tests xfail (#8186) 2022-08-25 21:04:05 +05:45
skshetry 54ee8c0d77 data status: make untracked files to be relative paths (#8184) 2022-08-25 19:29:43 +05:45
skshetry 063eb6904d data status: test cleanup and improvements (#8185) 2022-08-25 18:50:46 +05:45
David de la Iglesia Castro e815bbe211 metrics: show: Fix show from subdir.
This was a regression introduced in #7712.
Calling `repo.metrics.show` from a subdir stopped returning metrics because `fs.isfile(metric)` check returned false.
2022-08-25 13:02:22 +02:00
Saugat Pachhai (सौगात) 2374c72105 status: add unknown state 2022-08-25 13:23:18 +03:00
Saugat Pachhai (सौगात) 5cf1421c61 fix tests 2022-08-25 13:23:18 +03:00
Saugat Pachhai (सौगात) 6c1d00c2d9 fix diff when no object exists by falling back to shallow_diff 2022-08-25 13:23:18 +03:00
Saugat Pachhai (सौगात) f72456f67c add shallow_diff 2022-08-25 13:23:18 +03:00
skshetry 2c556f58d3 data status: fix not-in-cache hint (#8181)
See https://github.com/iterative/dvc.org/pull/3812\#discussion_r936625617.
2022-08-25 15:39:59 +05:45
Daniele Trifirò 26008f1559 import-url/update: add --no-download flag
add --no-download flag to dvc import-url/dvc update to only create/update .dvc
files without downloading the associated data.

Created .dvc files can be fetched using `dvc pull` or can be
updated using `dvc update --no-download`.
2022-08-23 23:07:01 +03:00
Daniele Trifirò 92d680ff94 tests: fix typo in test name 2022-08-23 23:07:01 +03:00
Ruslan Kuprieiev a79ea15827 tests: webhdfs: fix linter 2022-08-18 12:43:12 +03:00
Ruslan Kuprieiev 8db9a2f70f deps: bump dvc-data to 0.1.19 2022-08-18 12:43:12 +03:00
Ruslan Kuprieiev 3fc9f323a7 fs: don't import SSHFileSystem 2022-08-18 00:23:40 +03:00
Ruslan Kuprieiev 3b3866323e deps: bump dvc-data to 0.1.18 2022-08-17 23:02:22 +03:00
Ruslan Kuprieiev 3a429a869e tests: lockfile: use fs_path 2022-08-13 23:09:21 +03:00
Ruslan Kuprieiev fcdad4d229 deps: bump dvc-data to 0.1.16 2022-08-13 22:25:24 +03:00
Ivan Shcheklein 206e00ccbe guard rwlock file to prevent not locked repo edits 2022-08-11 15:54:42 +03:00
Peter Rowlands f13443477d exp run: account for repo root when staging git-deps 2022-08-11 13:48:28 +09:00
skshetry 7a0b3e9377 Update tests/func/experiments/test_queue.py 2022-08-09 17:50:48 +08:00
karajan1001 d6f8e5a507 Add flaky to a queue related test.
This is a flaky, the queue tasks status might remain "Running" even if
it had already been completed, beside sometimes I also noticed it raise
a pytest fixture error.
2022-08-09 17:50:48 +08:00
Ivan Shcheklein b7c0c41262 lock test name fix 2022-08-09 00:47:04 +03:00
karajan1001 9180a718c1 queue status: show final experiment name
fix: #8059

1. showing final experiment name in `queue status` for successful tasks.
2. Add a unit test for `celery_queue.status`
2022-08-08 15:49:51 +09:00
karajan1001 306456d453 queue status: ERROR: Invalid experiment
fix: #8014
> ERROR: Invalid experiment '{entry.stash_rev[:7]}'.

This happens when the queue task failed from a scm error.

{"exc_type": "GitMergeError", "exc_message": ["Cannot fast-forward HEAD
to '05100047a341f2fa4a02421289d48f84d8c45e86'"], "exc_module": "dvc.scm"}.

And the failed task neither create a infofile nor create a fail_stash.

1. Didn't raise DvcException if no infofile found for failed tasks.
   (success tasks still raise it)
2. Add a new unit test for this.
2022-08-08 15:07:11 +09:00
Ivan Shcheklein 95421a5605 fix logic for unlock on unlocked lock calls 2022-08-08 14:25:10 +09:00
David de la Iglesia Castro 31635b8312 exp run: Use hydra for parsing --set-param.
- Overriding a config value : foo.bar=value
- Appending a config value : +foo.bar=value
- Appending or overriding a config value : ++foo.bar=value
- Removing a config value : ~foo.bar, ~foo.bar=value

See https://hydra.cc/docs/advanced/override_grammar/basic/#modifying-the-config-object

---

Breaking changes:

To modify a list, `foo[0]=bar` must now be passed as `foo.0=bar`.

Modifying a nested list inside a dictionary is not supported by omegaconf.

---

Closes #4883
Closes #5868
Closes #6129
2022-08-05 17:53:24 +02:00
David de la Iglesia Castro 45ac64ddde api: Fix params_show when stage has no params
Also fix when `deps=True` and no params in the entire pipeline.

Closes #8064
2022-08-05 16:26:19 +03:00
Sami Abdul-Wahid c7b22b467c Upgrade from toml to tomlkit library (#7468)
* Upgrade from toml to tomlkit library
2022-08-05 18:58:22 +05:45
karajan1001 ae83e3c787 Now exp remove can clean up failed tasks and associated successful tasks
fix: #8018
fix: #8071

1. Add one line to make `exp remove` can remove failed tasks, and will
   also delete associated successful tasks
2. Add a new unit test for this behaviour.
3. Solve the unstable test_queue problem.
2022-08-05 16:09:51 +09:00
karajan1001 2182c4508d Make exp remove flags and arguments mutually exclusive.
1. Current behavior didn't follow the either or description.
2. Make the underlying logic more clear.
2022-08-05 16:09:51 +09:00
karajan1001 905c416ffc Decouple queue clear and remove commands 2022-08-05 16:09:51 +09:00
Ruslan Kuprieiev 0ce4e769f1 repo: make lock/rwlock kinda work on non local fs 2022-08-03 21:10:28 +03:00
Ruslan Kuprieiev fcc7240617 odb: use odb.path instead of odb.cache_dir 2022-08-03 21:10:28 +03:00
Ruslan Kuprieiev 6a7b55fdf7 datafs: use dvc-data's datafs 2022-08-02 03:13:06 +03:00
Ivan Shcheklein 9ff18502b6 add gdrive remote profile field to config 2022-07-27 23:29:18 +03:00
Paweł Redzyński 6543683c0c project: update dvc render and dvclive test dependencies 2022-07-26 13:40:59 +02:00
karajan1001 9dfe88e2b1 exp run: workspace status wrong
fix: #8045
2022-07-26 15:02:09 +05:45
Saugat Pachhai (सौगात) 896acedb07 data status: optimize printing status list
`--untracked-files` and `--unchanged` coupled with `--granular`
puts a lot of stress on `show_status`. Currently, with those options,
dvc spends ~26s on `_show_status` function.

Unhappy with the above result, I looked into improving the situation with
`console.out`. It did reduce that to 9s but it still felt like a lot.

Then I dropped to good old colorama, which dropped to 3s(term)/1.24(redirected).
I tested with buffering output which dropped this further to
~600ms(term)/~100ms(redirected).

This may seem like premature optimization, but this was preventing me
from figuring out perf issues in granular outputs, and I believe the
user should not be penalize for using feature that we already offer.
2022-07-25 17:47:23 +05:45
karajan1001 a4763593c7 queue: clean local EXEC_HEAD & EXEC_BASELINE after we run exp with queue.
fix: #8031

Currently when we use tmp directory to run a experiment, we set
`EXEC_HEAD` , `EXEC_BASELINE` and `EXEC_MERGE` heads locally but forget
to remove it. This cause some problem when we use `exp show`

1. Clean `EXEC_HEAD` , `EXEC_BASELINE` and `EXEC_MERGE` after we pushed
   them to tmp_dir
2. Add a unit test for it.

Co-authored-by:
2022-07-25 18:17:12 +08:00
Saugat Pachhai (सौगात) c3ac4f84bf data status: fix untracked files output for subdir 2022-07-22 16:29:04 +03:00
Saugat Pachhai (सौगात) 61729ff66a use M fixture for concise tests 2022-07-22 16:29:04 +03:00
Saugat Pachhai (सौगात) 3dfcb82e5c add tests for data:status command 2022-07-22 16:29:04 +03:00