karajan1001
e5b4946309
queue status: some reviewed changes.
...
1. No `start -j` will only start worker (1,2,3,...,j). Will not start
j new worker.
2. Distinguish active worker and idle worker.
2022-07-12 21:39:49 +09:00
karajan1001
3918db945a
queue status: will show the current worker status
...
fix : #7950
Currently queue worker status are not show in `queue status` makes it
hard to tell if the worker is still living.
1. add a new method `active_worker` to return the currently running
worker node name.
2. give every worker a unique name.
3. `queue status` will show how much worker running at present.
2022-07-12 21:39:49 +09:00
karajan1001
25b5d1d522
queue start: give every worker a unique name
...
Currently, all of our celery workers are called `dvc-exp-worker` and
they will write to the same log path.
We give every single worker a unique name and seperate they logs if
under a debug log level
2022-07-12 21:39:49 +09:00
dberenbaum
164efe089f
queue: ui cleanup
2022-07-12 21:39:49 +09:00
karajan1001
4e82eb4978
queue remove: add remove support for short revs( #7840 )
...
fix : #7840
Now queue remove can not accept short rev as name.
1. Refactor the queue entry match in `queue remove`
2. Better output if no queue task found for name.
3. short rev on kill can only be work for rev with lens longer than 4
2022-07-12 21:39:49 +09:00
karajan1001
16bb31a284
queue worker shutdown: disable restore
...
fix : #7884
The complete task restored back to the queue causing a task duplication.
Might related to a bug from the celery celery/celery#5663
Follow what in https://github.com/celery/celery/issues/5663#issuecomment-734871162
Disabled the restore_at_shutdown
2022-07-12 21:39:49 +09:00
karajan1001
d7a917e2ab
queue status: use a short rev to represent tasks
...
1. use a short rev to represent tasks
2. Add a unit test for the result shown
2022-07-12 21:39:49 +09:00
Peter Rowlands
8dfcb0a36d
exp run: update Ctrl+C handling for --temp and --run-all
2022-07-12 21:39:49 +09:00
Peter Rowlands
67cbdf6de8
exp queue: remove dvc queue attach command
2022-07-12 21:39:49 +09:00
Peter Rowlands
f12091002f
exp queue: add dvc queue logs command
2022-07-12 21:39:49 +09:00
Peter Rowlands
570f4345a0
exp queue: implement queue.logs() support
2022-07-12 21:39:49 +09:00
Peter Rowlands
425d0701c9
exp queue: return completed exps in queue status
2022-07-12 21:39:49 +09:00
Peter Rowlands
99a11f7056
exp: save failed queue runs in separate stash/reflog
2022-07-12 21:39:49 +09:00
Peter Rowlands
8ab46c1549
exp: simplify shutdown behavior
2022-07-12 21:39:49 +09:00
Peter Rowlands
8b663f2719
exp: use single atomic task instead of chaining
2022-07-12 21:39:49 +09:00
Peter Rowlands
c0db513cf0
dvc-task: bump to 0.0.11
2022-07-12 21:39:49 +09:00
Peter Rowlands
bf1bc9625d
exp: use celery async wait in queue get_result
2022-07-12 21:39:49 +09:00
Peter Rowlands
686c15dab9
exp: use a single chain for all exp tasks
...
- removes separated scheduling for setup_exp
2022-07-12 21:39:49 +09:00
Peter Rowlands
10aa833f80
dvc-task: bump to 0.0.10
2022-07-12 21:39:49 +09:00
Peter Rowlands
9f5af04963
exp: update tests for match_queue_entry refactor
2022-07-12 21:39:49 +09:00
Peter Rowlands
ac5d72967a
dvc-task: bump to 0.0.9
2022-07-12 21:39:49 +09:00
karajan1001
bcc2f3b389
Reorganized get_queue_entry_by_names
2022-07-12 21:39:49 +09:00
karajan1001
3d0ee0d202
exp queue: monitor queue status ( #7590 )
...
fix : #7590
1. Add two new command `queue status` and `queue attach`
2. Add unit test for the CLI
3. Make `Running` exps can be found by `get_queue_entry_by_names`
2022-07-12 21:39:49 +09:00
karajan1001
6b8c87b58a
Add more unit test and implement a new method
...
1. Add some more unit test for `celery_queue.kill`
2. Implement `kill` argument for `celery_queue.shutdown`
3. Use dataclass's internal methods instead of `__eq__`
2022-07-12 21:39:49 +09:00
Peter Rowlands
fee1d02791
dvc-task: update to 0.0.8
2022-07-12 21:39:49 +09:00
Peter Rowlands
d766c01d2d
exp: enable late_ack to disable celery prefetch
2022-07-12 21:39:49 +09:00
Peter Rowlands
dc56c4c9b0
exp: use full task chain when checking active tasks
2022-07-12 21:39:49 +09:00
Peter Rowlands
7ece26d58a
tests: re-enable exp show celery tests
2022-07-12 21:39:49 +09:00
Peter Rowlands
f26664a569
exp: re-enable run --run-all with celery queue
2022-07-12 21:39:49 +09:00
Peter Rowlands
4b0c65093a
tests: use pytest-celery for exp tests
2022-07-12 21:39:49 +09:00
karajan1001
12fc8c4e7d
exp queue: Add two command queue start and stop ( #7589 )
...
fix : #7589
1. Add two sub-command `dvc queue start` and `dvc queue stop`
2. Add a unit test to test them
2022-07-12 21:39:49 +09:00
karajan1001 and Peter Rowlands
a6cb4eb72e
exp queue: remove queued experiments( #7591 )
...
fix : #7591
1. Add a new command `dvc queue`
2. Add two sub-command `dvc queue remove` and `dvc queue kill`
3. Add a unit test to test them
Co-authored-by: Peter Rowlands (변기호) <[email protected]
2022-07-12 21:39:49 +09:00
karajan1001
4df5a1321d
exp: implement internal queue.kill support( #7587 )
...
fix : #7587
1. Add implement kill method for local queue class
2. Add a test to make sure the following job will be success after the
original job was killed.
3. Some refactoring work on `exp remove`
2022-07-12 21:39:49 +09:00
Peter Rowlands
22deca3a6a
exp: use celery threadpool worker instead of prefork
2022-07-12 21:39:49 +09:00
Peter Rowlands
dc4292c602
exp: implement queue shutdown
2022-07-12 21:39:49 +09:00
Peter Rowlands
56f90eb3c9
exp: add test_queue unit test helper for celery queue
2022-07-12 21:39:49 +09:00
Peter Rowlands
1c474e8f30
exp: re-enable most tmpdir/queue tests
2022-07-12 21:39:49 +09:00
Peter Rowlands
24654fe16c
exp: use dvc.daemon to run worker on windows
2022-07-12 21:39:49 +09:00
Peter Rowlands
44820e74a3
exp: wait for task collection when getting exp result
2022-07-12 21:39:49 +09:00
Peter Rowlands
c16908e9fc
exp: use queue remove in gc
2022-07-12 21:39:49 +09:00
Peter Rowlands
33dbdd20af
exp: exclude celery stash ref from util iterators
2022-07-12 21:39:49 +09:00
Peter Rowlands
69d7bdd0c0
exp: generate unique worker node names per DVC repo
2022-07-12 21:39:49 +09:00
Peter Rowlands
b9f4398826
exp: start migrating info logging to ui
2022-07-12 21:39:49 +09:00
Peter Rowlands
a6e7d43668
exp: return exp results for celery runs
2022-07-12 21:39:49 +09:00
Peter Rowlands
e2ee69ddb2
dvc-task: update to 0.0.6
2022-07-12 21:39:49 +09:00
Peter Rowlands
e8448345f1
exp: follow output on run-all
2022-07-12 21:39:49 +09:00
Peter Rowlands
acab008c65
dvc-task: update to 0.0.3
2022-07-12 21:39:49 +09:00
Peter Rowlands
8c74775a4e
exp: autostart celery worker
2022-07-12 21:39:49 +09:00
Peter Rowlands
9b31a4b463
exp: support remove from celery queue
2022-07-12 21:39:49 +09:00
Peter Rowlands
9f022c576c
dvc-task: update to 0.0.2
2022-07-12 21:39:49 +09:00
karajan1001
dberenbaum
Peter Rowlands