vLLM docs: KV Offloading Usage Guide (OffloadingConnector)

The mechanism and the bound: OffloadingConnector 'extends the prefix cache by offloading completed KV blocks to slower but larger tiers (CPU host memory, plus optional secondary tiers) as they are produced', and 'Transfers between GPU and CPU use DMA (cudaMemcpyAsync) and run asynchronously alongside model computation, so offloading adds minimal CPU- and GPU-core overhead'. Portability limit stated: 'The OffloadingConnector currently supports CUDA, ROCm, and XPU only.' Tier structure: CPUOffloadingSpec (single CPU tier, pinned host memory) vs TieringOffloadingSpec (CPU primary + secondary), where 'Only the CPU primary tier has direct GPU access' and all GPU-secondary transfers are staged through the CPU tier - a two-hop cost for anything below DRAM. Per-request caps max_load_tokens / max_offload_tokens; tokens beyond the load cap are recomputed rather than served.