Commit Graph
174 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 3fc9f323a7 fs: don't import SSHFileSystem 2022-08-18 00:23:40 +03:00
skshetry 7a0b3e9377 Update tests/func/experiments/test_queue.py 2022-08-09 17:50:48 +08:00
karajan1001 d6f8e5a507 Add flaky to a queue related test.
This is a flaky, the queue tasks status might remain "Running" even if
it had already been completed, beside sometimes I also noticed it raise
a pytest fixture error.
2022-08-09 17:50:48 +08:00
David de la Iglesia Castro 31635b8312 exp run: Use hydra for parsing --set-param.
- Overriding a config value : foo.bar=value
- Appending a config value : +foo.bar=value
- Appending or overriding a config value : ++foo.bar=value
- Removing a config value : ~foo.bar, ~foo.bar=value

See https://hydra.cc/docs/advanced/override_grammar/basic/#modifying-the-config-object

---

Breaking changes:

To modify a list, `foo[0]=bar` must now be passed as `foo.0=bar`.

Modifying a nested list inside a dictionary is not supported by omegaconf.

---

Closes #4883
Closes #5868
Closes #6129
2022-08-05 17:53:24 +02:00
karajan1001 905c416ffc Decouple queue clear and remove commands 2022-08-05 16:09:51 +09:00
Ruslan Kuprieiev fcc7240617 odb: use odb.path instead of odb.cache_dir 2022-08-03 21:10:28 +03:00
karajan1001 9dfe88e2b1 exp run: workspace status wrong
fix: #8045
2022-07-26 15:02:09 +05:45
David de la Iglesia Castro ed1d8e2b8f exp: init: Use persist: true if checkpoint. 2022-07-21 13:21:06 +02:00
Peter Rowlands 80272a0770 queue: cleanup flaky tests 2022-07-12 21:39:49 +09:00
karajan1001 02681f2e65 Rename _all to all_ remove some tricky func test. 2022-07-12 21:39:49 +09:00
karajan1001 b37524603a Queue remove: Can remove successed,failed tasks msgs
Followed from #7592

1. Seperate remove method to a new file.
2. Add queued, failed, processed flags to remove.
3. Add new unit tests for queue status --queue/fail/success
4. Implement methods to remove done tasks.
5. add some new unit and functional test for celery_queue.remove
6. bump into dvc-task version 0.0.13
2022-07-12 21:39:49 +09:00
Peter Rowlands 1f7b802f79 exp show: collect running standalone tempdir experiments 2022-07-12 21:39:49 +09:00
Peter Rowlands 9e0788aa3a exp run: preserve --temp standalone behavior 2022-07-12 21:39:49 +09:00
Peter Rowlands e1cf62a583 exp queue: add separate standalone/tempdir implementation 2022-07-12 21:39:49 +09:00
Peter Rowlands 570f4345a0 exp queue: implement queue.logs() support 2022-07-12 21:39:49 +09:00
Peter Rowlands 425d0701c9 exp queue: return completed exps in queue status 2022-07-12 21:39:49 +09:00
Peter Rowlands 99a11f7056 exp: save failed queue runs in separate stash/reflog 2022-07-12 21:39:49 +09:00
Peter Rowlands 7ece26d58a tests: re-enable exp show celery tests 2022-07-12 21:39:49 +09:00
Peter Rowlands f26664a569 exp: re-enable run --run-all with celery queue 2022-07-12 21:39:49 +09:00
Peter Rowlands 4b0c65093a tests: use pytest-celery for exp tests 2022-07-12 21:39:49 +09:00
Peter Rowlands 1c474e8f30 exp: re-enable most tmpdir/queue tests 2022-07-12 21:39:49 +09:00
Peter Rowlands 46c734eb7e exp: temporarily mark tmpdir/queue tests as xfail 2022-07-12 21:39:49 +09:00
Peter Rowlands 7113568c60 exp: refactor base.py into refs.py and exceptions.py 2022-07-12 21:39:49 +09:00
Peter Rowlands 6143146974 exp: refactor stash handling into separate module 2022-07-12 21:39:49 +09:00
David de la Iglesia Castro 20f69a121e api: Move to sub-package.
Split into sub-modules.

Split tests.
2022-06-21 20:52:19 +03:00
Peter Rowlands 31f67cd3de exp apply: preserve untracked files 2022-06-19 22:41:22 -07:00
dependabot[bot] bdea9847b5 build(deps-dev): Bump pylint from 2.13.9 to 2.14.1
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.13.9 to 2.14.1.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.13.9...v2.14.1)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2022-06-13 22:15:27 +05:45
dberenbaum 33b3afa2f5 exp init: create output dirs 2022-06-07 17:55:24 +05:45
David de la Iglesia Castro 34ef1807be exp: show: Include additional info in --json.
Use relpath to repo.root_dir as keys in deps and outs.
Add `use_cache` and `is_data_source` for outs.

This allows to differentiate git-tracked dependencies and
intermediate outputs.

Closes #7575
Closes #7790
2022-05-25 09:01:39 +02:00
Saugat Pachhai (सौगात) 570e3a5359 dvc.fs: move fs to dvc.objects.fs
Some of the APIs/FileSystems are reimported inside dvc.fs module.
2022-05-16 13:15:19 +03:00
Ruslan Kuprieiev 005d56eedf fs: rename RepoFileSystem to DvcFileSystem 2022-05-12 00:39:41 +03:00
Ruslan Kuprieiev a80a85e6bd dvc: detach gitfs from local paths
Our gitfs paths are currently tied to the local path in which the cloned
repository resides in, which results in a lot of hacky assumptions throughout
the code.

This PR detaches gitfs from local paths completely, making dvc work with a
proper fsspec gitfs implementation.

Part of #7543
2022-05-11 18:17:39 +03:00
dberenbaum 477f266dc5 exp init: fixes #7534; simplifies/updates exp init --live 2022-05-09 20:58:27 +05:45
karajan1001 c81d841fd7 exp pull/push: better result handling in exp sharing(#7448)
fix: #7448
Current we skip on those experiments which already existed on both sides, but
do not exclude them from the result list. And if we have diverged experiment,
pulling/pushing them will stop the current progress and raise an exception, but
didn't give any information on the condition of other experiments.

1. Change the exp pull and push to match the new API in `scmrepo`.
2. Handle sync status after `exp pull/push` finished.
3. Bump scmrepo to 0.0.19

Co-authored-by:
2022-04-28 19:56:15 +08:00
David de la Iglesia Castro 5c561572b0 exp init: Replace live section with metrics and plots.
Closes #7534
2022-04-21 08:55:49 +02:00
Ruslan Kuprieiev 9d75e8261b repofs: use repo paths 2022-04-18 14:17:20 +03:00
Peter Rowlands 69acbe33c9 erepo: include exp refs in clones
- add test for dvc get --rev=<exp>
2022-04-09 10:27:06 +09:00
David de la Iglesia Castro 117b22d58a exp show: pcp: Allow to --drop Experiment 2022-04-04 11:48:55 +02:00
Saugat Pachhai (सौगात) 7d0d6a38d7 exp init: change ui prompts structure and output 2022-03-29 16:20:32 +05:45
karajan1001andDave Berenbaum 325ead5e8b exp remove: show removed experiments #7449
fix: #7449
`exp remove`: should show the experiments that were removed

1. experiment.remove now returns the experiment list been removed.
2. modify the functional test to these the returned list.
3. use ui to show the removed list.

Co-authored-by: Dave Berenbaum <[email protected]>
2022-03-22 17:13:44 +08:00
Saugat Pachhai (सौगात) 9d96ddfbe7 exp init: create params file by default
Note that this won't add support for tracking the params file,
it'll only create the params file. Some changes on error and
prompts are made.
2022-03-21 15:00:20 +05:45
Saugat Pachhai (सौगात) e80e725fe4 tests: fix test failures with rich==12 2022-03-11 19:26:54 +05:45
karajan1001andJorge Orpinel efc021da73 exp remove: rename --all-commits flag and unify the collection of revs (#7155)
fix: #7155
1. rename flags `-A/--all-commits` in `exp remove`
2. add new flag `-n/--num` in `exp remove`
3. unify the revision collection in `exp remove`
4. add unit and func tests for `exp remove`

Co-authored-by: Jorge Orpinel <[email protected]>
2022-03-09 18:54:15 +08:00
karajan1001 244637e299 exp push/pull: add --all-commits flag and unify the collection of revs (#7154)
fix: #7154
1. rename flags `-A/--all-commits` in the `exp pull/push`
2. add new flag "--rev" and "--num" in the `exp pull/push`
3. Unify the collection of revs in `exp push/pull`
4. add unit and func tests for `exp push/pull`
2022-03-09 17:43:26 +08:00
karajan1001andJorge Orpinel d033345860 exp list: add the new flags (#7153)
fix: #7153
Unify the flags in `exp list` to `exp show`

1. Add new `-A`, `-n` flags to `exp list` command.
2. Add unit test for the flag parse.
3. unify the revs retrive logic to `exp show`.
4. Add functional test for `exp list`
5. Update the documents.

Co-authored-by: Jorge Orpinel <[email protected]>
2022-03-08 16:17:33 +08:00
Saugat Pachhai (सौगात) fcddf080da exp init: refactor and simplify interactive mode
- Moves workspace tree for interactive mode after prompts. Previously
  this was displayed before the prompts.
- Remove the yaml contents and the confirmation prompt in an interactive
  mode.
- Simplify `init_interactive()`.
2022-02-22 16:59:08 +05:45
Saugat Pachhai (सौगात) 497eac952c s/Creating/Created 2022-02-17 14:18:08 +05:45
Saugat Pachhai (सौगात) 6eb5c31dd7 fix UI message 2022-02-17 14:18:08 +05:45
Saugat Pachhai (सौगात) 69ce69e427 exp init: update message 2022-02-17 14:18:08 +05:45
Saugat Pachhai (सौगात) f8467cf375 exp init: log created dependencies in non-interactive mode 2022-02-17 14:18:08 +05:45