id
flops/ctx-fp8-kv-cache-long-context-accuracy
Arithmetic intensity
KV-cache dtype is the cleanest 2x in the whole stack: bytes per cached token halve from bf16's 2 bytes/element to fp8_e4m3's 1, so decode attention - which the repo already derives as memory-bound at AI = g (the GQA group ratio) flop/byte, independent of both batch and context length - reads half the bytes at the same flops, and decode AI doubles from g to 2g. Every consequence in this record follows from that one halving plus one non-arithmetic fact: with fp8 KV, vLLM's --kv-cache-dtype fp8 runs the ENTIRE attention computation (QK and ScoreV matmuls, and queries too under FA3) in the FP8 domain rather than only storing the cache compressed. That is a much bigger change than 'store the cache smaller' and it is where the long-context-specific failure lives. THE HOPPER ACCUMULATION BUG, which is the concrete long-context-specific degradation and the reason this record exists. Hopper's FP8 tensor cores are documented as accumulating into FP32 registers, but in practice the intermediate accumulation loses precision once the CONTRACTION DIMENSION is large. In the Softmax(AttnScore)*V matmul during long-context inference, the contraction dimension IS the context length. The measured consequence: on a 128k needle-in-a-haystack task, FP8 KV scored 13 percent against a 91 percent BF16 baseline - a 78-point collapse, with no error, no warning, and no change in the latency histogram. vLLM traced it to imprecise FP32 accumulation in the tensor cores, noting this is a known hardware-level issue also hit during DeepSeek-V3 training. The fix is two-level accumulation (flash-attention PR #104, following SageAttention2, arXiv 2411.10958), which writes partially accumulated results into a real FP32 register and brought 128k NIAH from 13 percent back to 89 percent. The trigger is explicit: the failure appears when the contraction dimension reaches 100K or more. SO THE DEGRADATION IS LENGTH-DEPENDENT BY CONSTRUCTION - this is the answer to 'does KV quantization degrade differently at long context than short'. It is not a smooth precision curve; there is a cliff at roughly 100K tokens that exists because of an arithmetic property of the matmul, and a workload at 8K or 32K cannot exhibit it no matter how badly its scales are chosen. The cost of the fix is prefill, not decode: two-level accumulation raises register pressure, and on gemma-4-E2B (head_dim 256) the TTFT quadratic coefficient rose from 6.93e-07 to 1.12e-06 ms/token^2, about 1.6x slower prefill. On head_dim 64 and 128 the optimized tile sizes recover most of that; on head_dim 256 the blog states prefill remains behind BF16. ON BLACKWELL THE BUG DOES NOT EXIST: on B200 with FlashInfer the accumulation issue is fixed and no two-level accumulation is needed - a hardware-generation difference in whether long-context FP8 attention is safe out of the box.
Scales with
context length (bytes per token halve, but the accumulation-precision failure turns on at ~100K contraction length, so the error term is a step function of L)head_dim (256 pays ~1.6x prefill from two-level accumulation; 64 and 128 mostly do not)whether layers are full-attention or sliding-window (small windows cannot amortize the overhead - skip them)attention backend (FA3 on Hopper needs two-level accumulation; FlashInfer on B200 does not; FlashMLA on MLA models is a third, less-validated path)tensor-parallel degree (cache shards, so the 2x capacity win applies per card)
Workarounds
Use fp8_e4m3 (fp8_e5m2 also exists in vLLM but has the smaller exponent range and no measured advantage here)Calibrate scales on your own data via llm-compressor before shipping uncalibrated fp8 to a non-standard backend; vLLM's docs call the uncalibrated per-tensor path the worst caseSkip sliding-window layers with --kv-cache-dtype-skip-layers sliding_window: for gpt-oss-20b this is the best variant, 6.34e-06 vs 7.14e-06 ms/token, and 871.8 vs 831.6 output tok/s at concurrency 8On Hopper at head_dim 256, either accept ~1.6x prefill or disable two-level accumulation AND validate accuracy yourself - the blog explicitly warns against the second option without extensive testingStay BF16 below about 7k tokens, where the intercept penalty is not yet amortized (break-even 7,010 tokens for Llama-3.1-8B on H100 after the fixes; 24,889 before them)
notes
MEASURED CAPACITY AND LATENCY WINS (all from the vLLM project blog of 2026-04-22, kind:blog, on their own software). DECODE COST, concurrency 1, single H100 with FA3 unless stated: Llama-3.1-8B fitted ITL slope falls from 4.37e-05 to 2.37e-05 ms/token - 54 percent of BF16, close to the 2x theoretical optimum, with the intercept essentially unchanged (6.44 -> 6.58 ms) and decode break-even around 7,010 tokens. Same model on B200 with FlashInfer: 1.80e-05 -> 9.72e-06 ms/token, also 54 percent, break-even around 4k. gpt-oss-20b (hybrid, sliding window 128) on H100: 8.94e-06 -> 7.14e-06 full fp8 (80 percent) and 6.34e-06 skipping sliding-window layers (71 percent), break-even 7,659 with skip-SW. gemma-4-E2B (head_dim 256, sliding window 512) on H100: 5.30e-05 -> 3.60e-05 (68 percent). The blog's summary claim is that in the best cases the per-token KV cost falls to 54 percent of its bf16 counterpart, not 50 - the gap is the fixed per-token quantization overhead. END-TO-END at concurrency 8, 150 requests of roughly 20K in and 2K out, which is the number that shows the memory win becoming scheduling win: Llama-3.1-8B BF16 763.6 ms median TTFT / 15.18 ms median ITL / 450.3 output tok/s versus FP8 742.8 / 12.93 / 517.5 - so +14.9 percent output throughput and -14.8 percent median ITL, and the stated mechanism is that the 2x memory reduction lets the scheduler pack more concurrent requests, not just decode faster. gpt-oss-20b gains only 4.8 percent because sliding-window layers cap the memory saving. ACCURACY ON REAL LONG-CONTEXT EVALUATIONS, using openai/mrcr with UNCALIBRATED per-tensor scales (the worst case by construction, so these are lower bounds): Llama-3.3-70B-Instruct to 128K recovers 97-98 percent of BF16 AUC; Qwen3-30B-A3B-Instruct-2507 to 256K recovers about 94 percent with bf16 weights and about 98 percent with fp8 weights; Qwen3.5-27B to 1M fully recovers aggregate AUC, with variance only in the longest buckets. Reasoning benchmarks (AIME25, GPQA:Diamond, MATH500, LiveCodeBench-v6) move by at most 1-2 points, worst recovery 97 percent (GPQA:Diamond, bf16 model); Qwen3.5-27B at most 0.7 points, worst recovery 99 percent. On B200 with FlashInfer, Qwen3-30B-A3B recovers about 93 percent (bf16 weights) to 96 percent (fp8 weights) - good but not uniformly as tight as the best Hopper FA3 cases. THE ONE MODEL WHERE THE BLOG RECOMMENDS CALIBRATING: Kimi-K2.5 on the FlashMLA backend with uncalibrated fp8 on H200 shows a consistently negative shift across context-length buckets - modest in aggregate AUC, with overlapping standard-error bands, but systematically rather than randomly negative. That is a backend-specific finding and it is the honest boundary of the 'fp8 KV is free' folklore. WHAT THIS DOES NOT SETTLE: the repo's kivi-kv2 and kvquant records both report quality deltas from perplexity on Wikitext-2 and C4, which are SHORT-context measures - KVQuant's headline is '<0.1 perplexity degradation with 3-bit quantization on both Wikitext-2 and C4'. Neither paper's quoted number is evidence about 128K behaviour, and this record is the correction: perplexity at 2048 tokens cannot see the 100K contraction cliff, cannot see the accumulation bug, and cannot see the head_dim 256 prefill regression. INT8 KV is the near-identical story with no separate measurement here - vLLM documents kv_cache_dtype fp8/int8 as supported and the repo's qhw-int8-kv-cache record carries quality_delta: None. PROVENANCE WARNING: the vLLM blog is first-party on their own engine, dated 2026-04-22, and several of the models it evaluates (Qwen3.5-27B, gemma-4, Kimi-K2.5) postdate most of this repo's other sources; its accuracy claims are self-reported and its performance claims are on configurations the authors tuned.