Peter Rowlands
3130a9a75f
queue status: use summary style output for worker status
2022-07-12 21:39:49 +09:00
Peter Rowlands
eeb516e7d3
queue: disable dvc-machine handling for queue
2022-07-12 21:39:49 +09:00
karajan1001
c0368e1756
Refactor: Move remove_revs from BaseStashQueue to expstash
2022-07-12 21:39:49 +09:00
Gao and Peter Rowlands
e19824fd2e
Update dvc/repo/experiments/queue/base.py dvc/repo/experiments/queue/remove.py
...
Co-authored-by: Peter Rowlands (변기호) <[email protected]
2022-07-12 21:39:49 +09:00
karajan1001
02681f2e65
Rename _all to all_ remove some tricky func test.
2022-07-12 21:39:49 +09:00
karajan1001
b37524603a
Queue remove: Can remove successed,failed tasks msgs
...
Followed from #7592
1. Seperate remove method to a new file.
2. Add queued, failed, processed flags to remove.
3. Add new unit tests for queue status --queue/fail/success
4. Implement methods to remove done tasks.
5. add some new unit and functional test for celery_queue.remove
6. bump into dvc-task version 0.0.13
2022-07-12 21:39:49 +09:00
Peter Rowlands
ff3b23c1b6
dvc-task: bump to 0.1.0
...
- ends requirement to install kombu + dvc-task from source
2022-07-12 21:39:49 +09:00
Peter Rowlands
4e005411fe
exp show: use unique executor name for dvc-task & standalone tempdir exps
2022-07-12 21:39:49 +09:00
Peter Rowlands
1f7b802f79
exp show: collect running standalone tempdir experiments
2022-07-12 21:39:49 +09:00
Peter Rowlands
7c2bd340db
exp run: update --run-all logs behavior
2022-07-12 21:39:49 +09:00
Peter Rowlands
9e0788aa3a
exp run: preserve --temp standalone behavior
2022-07-12 21:39:49 +09:00
Peter Rowlands
e1cf62a583
exp queue: add separate standalone/tempdir implementation
2022-07-12 21:39:49 +09:00
Peter Rowlands
ce4cee184e
exp queue: separate queue implementations into their own modules
2022-07-12 21:39:49 +09:00
Peter Rowlands
db52c75221
dvc-task: bump to 0.0.12
2022-07-12 21:39:49 +09:00
karajan1001 and Peter Rowlands
bd74172633
queue status: add some unit test to increase the coverage
...
Followed by #7903
Add a new unit test to increase the coverage of test
Co-authored-by: Peter Rowlands (변기호) <[email protected]
2022-07-12 21:39:49 +09:00
karajan1001
e5b4946309
queue status: some reviewed changes.
...
1. No `start -j` will only start worker (1,2,3,...,j). Will not start
j new worker.
2. Distinguish active worker and idle worker.
2022-07-12 21:39:49 +09:00
karajan1001
3918db945a
queue status: will show the current worker status
...
fix : #7950
Currently queue worker status are not show in `queue status` makes it
hard to tell if the worker is still living.
1. add a new method `active_worker` to return the currently running
worker node name.
2. give every worker a unique name.
3. `queue status` will show how much worker running at present.
2022-07-12 21:39:49 +09:00
karajan1001
25b5d1d522
queue start: give every worker a unique name
...
Currently, all of our celery workers are called `dvc-exp-worker` and
they will write to the same log path.
We give every single worker a unique name and seperate they logs if
under a debug log level
2022-07-12 21:39:49 +09:00
dberenbaum
164efe089f
queue: ui cleanup
2022-07-12 21:39:49 +09:00
karajan1001
4e82eb4978
queue remove: add remove support for short revs( #7840 )
...
fix : #7840
Now queue remove can not accept short rev as name.
1. Refactor the queue entry match in `queue remove`
2. Better output if no queue task found for name.
3. short rev on kill can only be work for rev with lens longer than 4
2022-07-12 21:39:49 +09:00
karajan1001
16bb31a284
queue worker shutdown: disable restore
...
fix : #7884
The complete task restored back to the queue causing a task duplication.
Might related to a bug from the celery celery/celery#5663
Follow what in https://github.com/celery/celery/issues/5663#issuecomment-734871162
Disabled the restore_at_shutdown
2022-07-12 21:39:49 +09:00
karajan1001
d7a917e2ab
queue status: use a short rev to represent tasks
...
1. use a short rev to represent tasks
2. Add a unit test for the result shown
2022-07-12 21:39:49 +09:00
Peter Rowlands
8dfcb0a36d
exp run: update Ctrl+C handling for --temp and --run-all
2022-07-12 21:39:49 +09:00
Peter Rowlands
67cbdf6de8
exp queue: remove dvc queue attach command
2022-07-12 21:39:49 +09:00
Peter Rowlands
f12091002f
exp queue: add dvc queue logs command
2022-07-12 21:39:49 +09:00
Peter Rowlands
570f4345a0
exp queue: implement queue.logs() support
2022-07-12 21:39:49 +09:00
Peter Rowlands
425d0701c9
exp queue: return completed exps in queue status
2022-07-12 21:39:49 +09:00
Peter Rowlands
99a11f7056
exp: save failed queue runs in separate stash/reflog
2022-07-12 21:39:49 +09:00
Peter Rowlands
8ab46c1549
exp: simplify shutdown behavior
2022-07-12 21:39:49 +09:00
Peter Rowlands
8b663f2719
exp: use single atomic task instead of chaining
2022-07-12 21:39:49 +09:00
Peter Rowlands
c0db513cf0
dvc-task: bump to 0.0.11
2022-07-12 21:39:49 +09:00
Peter Rowlands
bf1bc9625d
exp: use celery async wait in queue get_result
2022-07-12 21:39:49 +09:00
Peter Rowlands
686c15dab9
exp: use a single chain for all exp tasks
...
- removes separated scheduling for setup_exp
2022-07-12 21:39:49 +09:00
Peter Rowlands
10aa833f80
dvc-task: bump to 0.0.10
2022-07-12 21:39:49 +09:00
Peter Rowlands
9f5af04963
exp: update tests for match_queue_entry refactor
2022-07-12 21:39:49 +09:00
Peter Rowlands
ac5d72967a
dvc-task: bump to 0.0.9
2022-07-12 21:39:49 +09:00
karajan1001
bcc2f3b389
Reorganized get_queue_entry_by_names
2022-07-12 21:39:49 +09:00
karajan1001
3d0ee0d202
exp queue: monitor queue status ( #7590 )
...
fix : #7590
1. Add two new command `queue status` and `queue attach`
2. Add unit test for the CLI
3. Make `Running` exps can be found by `get_queue_entry_by_names`
2022-07-12 21:39:49 +09:00
karajan1001
6b8c87b58a
Add more unit test and implement a new method
...
1. Add some more unit test for `celery_queue.kill`
2. Implement `kill` argument for `celery_queue.shutdown`
3. Use dataclass's internal methods instead of `__eq__`
2022-07-12 21:39:49 +09:00
Peter Rowlands
fee1d02791
dvc-task: update to 0.0.8
2022-07-12 21:39:49 +09:00
Peter Rowlands
d766c01d2d
exp: enable late_ack to disable celery prefetch
2022-07-12 21:39:49 +09:00
Peter Rowlands
dc56c4c9b0
exp: use full task chain when checking active tasks
2022-07-12 21:39:49 +09:00
Peter Rowlands
7ece26d58a
tests: re-enable exp show celery tests
2022-07-12 21:39:49 +09:00
Peter Rowlands
f26664a569
exp: re-enable run --run-all with celery queue
2022-07-12 21:39:49 +09:00
Peter Rowlands
4b0c65093a
tests: use pytest-celery for exp tests
2022-07-12 21:39:49 +09:00
karajan1001
12fc8c4e7d
exp queue: Add two command queue start and stop ( #7589 )
...
fix : #7589
1. Add two sub-command `dvc queue start` and `dvc queue stop`
2. Add a unit test to test them
2022-07-12 21:39:49 +09:00
karajan1001 and Peter Rowlands
a6cb4eb72e
exp queue: remove queued experiments( #7591 )
...
fix : #7591
1. Add a new command `dvc queue`
2. Add two sub-command `dvc queue remove` and `dvc queue kill`
3. Add a unit test to test them
Co-authored-by: Peter Rowlands (변기호) <[email protected]
2022-07-12 21:39:49 +09:00
karajan1001
4df5a1321d
exp: implement internal queue.kill support( #7587 )
...
fix : #7587
1. Add implement kill method for local queue class
2. Add a test to make sure the following job will be success after the
original job was killed.
3. Some refactoring work on `exp remove`
2022-07-12 21:39:49 +09:00
Peter Rowlands
22deca3a6a
exp: use celery threadpool worker instead of prefork
2022-07-12 21:39:49 +09:00
Peter Rowlands
dc4292c602
exp: implement queue shutdown
2022-07-12 21:39:49 +09:00
Peter Rowlands
karajan1001
dberenbaum