The State of FP8 KV-Cache and Attention Quantization in vLLM (vLLM project blog, 2026-04-22)

First-party engineering validation from the engine authors, on their own software, with named models and hardware. Marked kind:blog because it is not a peer-reviewed result, but it is the only source located that measures FP8 KV quantization against LONG-CONTEXT accuracy specifically rather than short-context perplexity. Key numbers. ACCURACY AT LONG CONTEXT: on a 128k needle-in-a-haystack task on Hopper, FP8 KV with the FA3 backend scored 13% vs a 91% BF16 baseline - traced to imprecise FP32 accumulation in Hopper's FP8 tensor cores when the contraction dimension (which in the Softmax(AttnScore)*V matmul equals the CONTEXT LENGTH) reaches 100K or more. A two-level accumulation strategy (flash-attention PR #104, after SageAttention2 arXiv 2411.10958) restoring a real FP32 register brought it to 89%. Cost of the fix: increased register pressure, so prefill slows; TTFT quadratic coefficient on gemma-4-E2B (head_dim 256) rose from 6.93e-07 to 1.12e-06 ms/token^2, i.e. ~1.6x slower prefill at long context. On B200 with FlashInfer the accumulation issue does not occur and no two-level accumulation is needed. ACCURACY ON REAL LONG-CONTEXT BENCHMARKS (openai/mrcr, uncalibrated per-tensor scales, i.e. the worst case): Llama-3.3-70B-Instruct to 128K recovers 97-98% of BF16 AUC; Qwen3-30B-A3B-Instruct-2507 to 256K recovers ~94% (BF16 weights) to ~98% (FP8 weights); Qwen3.5-27B to 1M fully recovers aggregate AUC. Reasoning benchmarks (AIME25, GPQA:Diamond, MATH500, LiveCodeBench-v6) shift by at most 1-2 points, worst recovery 97% (GPQA:Diamond, BF16 model); Qwen3.5-27B at most 0.7 points, worst recovery 99%. Kimi-K2.5 on the FlashMLA backend with uncalibrated FP8 on H200 shows a systematically negative shift across context-length buckets - modest in aggregate AUC but consistent, i.e. the one case where calibration is called for. DECODE COST: fitted ITL slope ITL = slope*input_len + intercept at concurrency 1. Llama-3.1-8B on H100/FA3: slope 4.37e-05 -> 2.37e-05 ms/token (54% of BF16), intercept 6.44 -> 6.58 ms, decode break-even ~7,010 tokens (was 24,889 before the fixes). Llama-3.1-8B on B200/FlashInfer: 1.80e-05 -> 9.72e-06 ms/token (54%), break-even ~4k. gpt-oss-20b (sliding window 128) on H100: 8.94e-06 -> 7.14e-06 full FP8 (80%) and 6.34e-06 skipping sliding-window layers (71%), break-even 7,659 tokens with skip-SW. On B200 gpt-oss-20b: 3.56e-06 -> 2.06e-06, break-even ~13k. gemma-4-E2B (head_dim 256, sliding window 512): 5.30e-05 -> 3.60e-05 (68%). Best-case KV memory per cached token falls to 54% of its BF16 counterpart. THROUGHPUT at concurrency 8, 150 requests of ~20K in / ~2K out: Llama-3.1-8B BF16 763.6 ms median TTFT, 15.18 ms median ITL, 450.3 output tok/s vs FP8 742.8 ms, 12.93 ms, 517.5 tok/s (+14.9% output throughput, -14.8% median ITL). gpt-oss-20b: BF16 831.6 tok/s vs FP8 853.0 vs FP8 skip-SW 871.8 tok/s. GUIDANCE: stay in BF16 if context < ~7k, if head_dim 256 and prefill latency matters, if uncalibrated accuracy recovery is under 95% on your workload, or if the model has many small sliding-window layers. Flag: several of these models and versions (Qwen3.5, gemma-4, Kimi-K2.5) are newer than most of this repo's other sources.