vLLM PR #40459: RFC - Rust-accelerated control plane (scheduler + KV cache manager), PoC
repo
Fields
id
sources/flop3-vllm-pr-40459-rust-control-planePublisher
vllm-project/vllm
Kind
repo
Accessed
2026-10-03
notes
Iteration-level Python overhead measured, not asserted. Qwen3-0.6B on H100 with enforce_eager=True and num_seqs=256, 12.68 s wall: Scheduler.schedule() 907 ms = 7.2%; Scheduler.update_from_output() 572 ms = 4.8%; total scheduler CPU 11.7% of wall. vLLM's own code flags the second loop at scheduler.py:1340 - 'As len(num_scheduled_tokens) can be up to 1K or more, the below loop can be a performance bottleneck.' Users report scenarios with 4 ms CPU / 1.4 ms GPU per step, i.e. the CPU plane ~3x the GPU work, and the PR states the overhead fraction is much larger on CPU-heavy workloads (small models, multimodal/TTS, higher batch, lower-end GPUs). Marked open / PoC, not a merge proposal.
Notes
Iteration-level Python overhead measured, not asserted. Qwen3-0.6B on H100 with enforce_eager=True and num_seqs=256, 12.68 s wall: Scheduler.schedule() 907 ms = 7.2%; Scheduler.update_from_output() 572 ms = 4.8%; total scheduler CPU 11.7% of wall. vLLM's own code flags the second loop at scheduler.py:1340 - 'As len(num_scheduled_tokens) can be up to 1K or more, the below loop can be a performance bottleneck.' Users report scenarios with 4 ms CPU / 1.4 ms GPU per step, i.e. the CPU plane ~3x the GPU work, and the PR states the overhead fraction is much larger on CPU-heavy workloads (small models, multimodal/TTS, higher batch, lower-end GPUs). Marked open / PoC, not a merge proposal.
Cross-references
This record references no other records.