Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,743 Commits 7 Branches 579 Tags
dd32ff62069776e9ab30c32907235bfcaf65b31d
Commit Graph
7743 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev dd32ff6206 deps: bump dvc-data to 0.0.31 2022-07-19 04:54:34 +03:00
Ruslan Kuprieiev 52a9c58cc0 deps: bump dvc-data to 0.0.30 2022-07-19 04:36:41 +03:00
Ruslan Kuprieiev 01855b7dcf deps: bump dvc-data to 0.0.28 2022-07-18 21:46:22 +03:00
Ruslan Kuprieiev 38f10884a7 deps: bump dvc-data to 0.0.27 2022-07-18 19:15:02 +03:00
Ruslan Kuprieiev 9e6d2163df deps: bump dvc-data to 0.0.26 2022-07-18 01:03:06 +03:00
skshetry dbfe88510f deps: bump dvc-data to 0.0.24 2022-07-15 22:40:26 +05:45
Alexander Guschin c04eb40f90 Update setup.cfg 2022-07-14 18:12:47 +05:45
David de la Iglesia Castro c2668110b7 gc: Add --rev and --num. 
Closes #7695
 2022-07-12 20:13:07 +02:00
Peter Rowlands 80272a0770 queue: cleanup flaky tests 2022-07-12 21:39:49 +09:00
Peter Rowlands 5a6e363c16 queue: cleanup help/usage to match docs 2022-07-12 21:39:49 +09:00
Peter Rowlands ed593493df queue start: simply jobs validation 2022-07-12 21:39:49 +09:00
Peter Rowlands 3130a9a75f queue status: use summary style output for worker status 2022-07-12 21:39:49 +09:00
Peter Rowlands eeb516e7d3 queue: disable dvc-machine handling for queue 2022-07-12 21:39:49 +09:00
karajan1001 c0368e1756 Refactor: Move remove_revs from BaseStashQueue to expstash 2022-07-12 21:39:49 +09:00
GaoandPeter Rowlands e19824fd2e Update dvc/repo/experiments/queue/base.py dvc/repo/experiments/queue/remove.py 
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
 2022-07-12 21:39:49 +09:00
karajan1001 02681f2e65 Rename _all to all_ remove some tricky func test. 2022-07-12 21:39:49 +09:00
karajan1001 b37524603a Queue remove: Can remove successed,failed tasks msgs 
Followed from #7592

1. Seperate remove method to a new file.
2. Add queued, failed, processed flags to remove.
3. Add new unit tests for queue status --queue/fail/success
4. Implement methods to remove done tasks.
5. add some new unit and functional test for celery_queue.remove
6. bump into dvc-task version 0.0.13
 2022-07-12 21:39:49 +09:00
Peter Rowlands ff3b23c1b6 dvc-task: bump to 0.1.0 
- ends requirement to install kombu + dvc-task from source
 2022-07-12 21:39:49 +09:00
Peter Rowlands 4e005411fe exp show: use unique executor name for dvc-task & standalone tempdir exps 2022-07-12 21:39:49 +09:00
Peter Rowlands 1f7b802f79 exp show: collect running standalone tempdir experiments 2022-07-12 21:39:49 +09:00
Peter Rowlands 7c2bd340db exp run: update --run-all logs behavior 2022-07-12 21:39:49 +09:00
Peter Rowlands 9e0788aa3a exp run: preserve --temp standalone behavior 2022-07-12 21:39:49 +09:00
Peter Rowlands e1cf62a583 exp queue: add separate standalone/tempdir implementation 2022-07-12 21:39:49 +09:00
Peter Rowlands ce4cee184e exp queue: separate queue implementations into their own modules 2022-07-12 21:39:49 +09:00
Peter Rowlands db52c75221 dvc-task: bump to 0.0.12 2022-07-12 21:39:49 +09:00
karajan1001andPeter Rowlands bd74172633 queue status: add some unit test to increase the coverage 
Followed by #7903

Add a new unit test to increase the coverage of test

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
 2022-07-12 21:39:49 +09:00
karajan1001 e5b4946309 queue status: some reviewed changes. 
1. No `start -j` will only start worker (1,2,3,...,j). Will not start
   j new worker.
2. Distinguish active worker and idle worker.
 2022-07-12 21:39:49 +09:00
karajan1001 3918db945a queue status: will show the current worker status 
fix: #7950
Currently queue worker status are not show in `queue status` makes it
hard to tell if the worker is still living.

1. add a new method `active_worker` to return the currently running
   worker node name.
2. give every worker a unique name.
3. `queue status` will show how much worker running at present.
 2022-07-12 21:39:49 +09:00
karajan1001 25b5d1d522 queue start: give every worker a unique name 
Currently, all of our celery workers are called `dvc-exp-worker` and
they will write to the same log path.

We give every single worker a unique name and seperate they logs if
under a debug log level
 2022-07-12 21:39:49 +09:00
dberenbaum 164efe089f queue: ui cleanup 2022-07-12 21:39:49 +09:00
karajan1001 4e82eb4978 queue remove: add remove support for short revs(#7840) 
fix: #7840

Now queue remove can not accept short rev as name.

1. Refactor the queue entry match in `queue remove`
2. Better output if no queue task found for name.
3. short rev on kill can only be work for rev with lens longer than 4
 2022-07-12 21:39:49 +09:00
karajan1001 16bb31a284 queue worker shutdown: disable restore 
fix: #7884
The complete task restored back to the queue causing a task duplication.
Might related to a bug from the celery celery/celery#5663

Follow what in https://github.com/celery/celery/issues/5663#issuecomment-734871162
Disabled the restore_at_shutdown
 2022-07-12 21:39:49 +09:00
karajan1001 d7a917e2ab queue status: use a short rev to represent tasks 
1. use a short rev to represent tasks
2. Add a unit test for the result shown
 2022-07-12 21:39:49 +09:00
Peter Rowlands 8dfcb0a36d exp run: update Ctrl+C handling for --temp and --run-all 2022-07-12 21:39:49 +09:00
Peter Rowlands 67cbdf6de8 exp queue: remove dvc queue attach command 2022-07-12 21:39:49 +09:00
Peter Rowlands f12091002f exp queue: add dvc queue logs command 2022-07-12 21:39:49 +09:00
Peter Rowlands 570f4345a0 exp queue: implement queue.logs() support 2022-07-12 21:39:49 +09:00
Peter Rowlands 425d0701c9 exp queue: return completed exps in queue status 2022-07-12 21:39:49 +09:00
Peter Rowlands 99a11f7056 exp: save failed queue runs in separate stash/reflog 2022-07-12 21:39:49 +09:00
Peter Rowlands 8ab46c1549 exp: simplify shutdown behavior 2022-07-12 21:39:49 +09:00
Peter Rowlands 8b663f2719 exp: use single atomic task instead of chaining 2022-07-12 21:39:49 +09:00
Peter Rowlands c0db513cf0 dvc-task: bump to 0.0.11 2022-07-12 21:39:49 +09:00
Peter Rowlands bf1bc9625d exp: use celery async wait in queue get_result 2022-07-12 21:39:49 +09:00
Peter Rowlands 686c15dab9 exp: use a single chain for all exp tasks 
- removes separated scheduling for setup_exp
 2022-07-12 21:39:49 +09:00
Peter Rowlands 10aa833f80 dvc-task: bump to 0.0.10 2022-07-12 21:39:49 +09:00
Peter Rowlands 9f5af04963 exp: update tests for match_queue_entry refactor 2022-07-12 21:39:49 +09:00
Peter Rowlands ac5d72967a dvc-task: bump to 0.0.9 2022-07-12 21:39:49 +09:00
karajan1001 bcc2f3b389 Reorganized get_queue_entry_by_names 2022-07-12 21:39:49 +09:00
karajan1001 3d0ee0d202 exp queue: monitor queue status (#7590) 
fix: #7590
1. Add two new command `queue status` and `queue attach`
2. Add unit test for the CLI
3. Make `Running` exps can be found by `get_queue_entry_by_names`
 2022-07-12 21:39:49 +09:00
karajan1001 6b8c87b58a Add more unit test and implement a new method 
1. Add some more unit test for `celery_queue.kill`
2. Implement `kill` argument for `celery_queue.shutdown`
3. Use dataclass's internal methods instead of `__eq__`
 2022-07-12 21:39:49 +09:00