Inside vLLM's New KV Offloading Connector (vLLM blog, 2026-01-08)

Vendor-authored but with full methodology, which is why it is usable. The stated win condition is the interesting part: offload 'can be useful even for workloads where requests share no common prefix' because when many concurrent requests exhaust GPU KV the engine preempts a request and would otherwise recompute its KV; offloading before preemption avoids the recompute. That is the long-context / high-batch-pressure win, not the low-batch case. Claim: loading KV from CPU 'reduces TTFT by X2-X22' depending on prompt size, while offload (GPU->CPU) is asynchronous and 'has minimal effect on TTFT for cache misses'. Setup given: H100 80GB HBM3, 500GB DRAM, CUDA 12.9, vLLM commit 2a1776b7ac4fae7c50c694edeafc1b14270e4350, GPU block size 16 tokens, GPU prefix caching disabled, de/tokenization off. Feature landed in vLLM 0.11.0; the async connector API it uses was added in 0.9.0.