inference/research
verifiedmedium · 0.80updated 2026-10-04

KV cache at the claimed context length, and how many 128K-context users one H100 or H200 can actually serve

memory bound

Fields

id
flops/ctx-kv-cache-gib-at-claimed-context
Class
kv_transfer
Bound by
memory
Arithmetic intensity
KV bytes per token is exact arithmetic with no measurement in it: B_per_token = n_attn_layers * n_kv_heads * head_dim * 2 (K and V) * bytes_per_element, with bytes_per_element = 2 for bf16, 1 for fp8_e4m3 or int8, 0.5 for fp4. Two cases need care. MLA does not use that formula: DeepSeek V2/V3/V3.2 and Kimi-K2 store a latent cache of kv_lora_rank 512 + qk_rope_head_dim 64 = 576 elements per layer, so B_per_token = n_layers * 576 * bytes_per_element = 61*576*2 = 70,272 B = 68.6 KiB, and there is no num_key_value_heads term at all - num_key_value_heads is 128 in the config and irrelevant to cache size. Qwen3-Next is a hybrid: full_attention_interval 4 over 48 layers means 12 attention layers and 36 gated-deltanet linear-attention layers, so only 12 layers contribute KV: 12 * 2 * 256 * 2 * 2 = 24,576 B = 24.0 KiB per token, the cheapest full-attention cache in this set. Everything else in the table is dense GQA or MQA and uses the first formula. BYTES PER TOKEN AND THE TWO LENGTHS THAT MATTER (all bf16, computed from config.json in [[ctx-src-hf-configs-longctx-models]]): Llama-3.1-8B 131,072 B/tok = 128.0 KiB, 16.00 GiB at 128K, 122.07 GiB at 1M. Llama-3.1-70B 327,680 B = 320.0 KiB, 40.00 GiB at 128K, 305.18 GiB at 1M. Llama-3.1-405B 516,096 B = 504.0 KiB, 63.00 GiB at 128K, 480.65 GiB at 1M. Qwen2.5-7B and Qwen2.5-7B-Instruct-1M (identical geometry) 57,344 B = 56.0 KiB, 7.00 GiB at 128K, 53.41 GiB at 1M. Qwen2.5-72B 327,680 B = 320.0 KiB, 40.00 GiB at 128K. Qwen3-8B 147,456 B = 144.0 KiB, 18.00 GiB at 128K, 137.33 GiB at 1M. Qwen3-235B-A22B 192,512 B = 188.0 KiB, 23.50 GiB at 128K, 179.29 GiB at 1M. Qwen3-Next-80B-A3B 24,576 B = 24.0 KiB, 3.00 GiB at 128K, 22.89 GiB at 1M. Mistral-Large-2411 360,448 B = 352.0 KiB, 44.00 GiB at 128K, 335.69 GiB at 1M. Command-R-plus-0824 262,144 B = 256.0 KiB, 32.00 GiB at 128K, 244.14 GiB at 1M. Yi-34B-200K 245,760 B = 240.0 KiB, 30.00 GiB at 128K, 228.88 GiB at 1M. GLM-4.9B-chat-1M 81,920 B = 80.0 KiB, 10.00 GiB at 128K, 76.29 GiB at 1M. ChatGLM3-6B-128K 32,768 B = 32.0 KiB, 4.00 GiB at 128K. GLM-4.5 376,832 B = 368.0 KiB, 46.00 GiB at 128K, 350.95 GiB at 1M. DeepSeek-V2-Chat 69,120 B = 67.5 KiB (60 layers), 8.44 GiB at 128K. DeepSeek-V3 / Kimi-K2 70,272 B = 68.6 KiB, 8.58 GiB at 128K, 4.29 GiB in fp8, 2.14 GiB in fp4. THE HEADLINE: at 128K, a 128K-token sequence's KV cache alone ranges from 3.00 GiB (Qwen3-Next hybrid) to 63.00 GiB (Llama-3.1-405B) across models a practitioner might choose for the same job - a 21x spread driven entirely by config, and a second-order effect next to how many of them fit on the card at all.
Scales with
context length (linear in tokens, and that is the whole problem: doubling context doubles the cache and halves concurrency)num_kv_heads and head_dim and n_attn_layers (the exact product, so GQA ratio and layer count are the two levers that matter)kv_cache_dtype (2 bytes bf16 -> 1 byte fp8/int8 -> 0.5 byte fp4; a 4x spread between bf16 and fp4)tensor-parallel degree (the cache shards by TP, so TP buys concurrency back at the cost of per-layer all-reduce; a 70B at TP4 puts 80 KiB/token on each card instead of 320)
Workarounds
Serve the MEASURED effective length, not the claimed one: on Llama-3.1-70B, capping at 64K instead of 128K doubles concurrency outright and costs nothing the model could reliably do anywayFP8 or INT8 KV halves the cache and roughly doubles concurrency at no accuracy cost on the validated paths ([[ctx-fp8-kv-cache-long-context-accuracy]])Prefer GQA/MQA or MLA or hybrid-interval architectures over MHA when the context claim is fixed: Qwen3-Next's 24 KiB/token is 21x cheaper than Llama-3.1-405B's 504 KiB/token at the same 128K lengthRaise tensor-parallel degree to shard the cache, accepting the all-reduce cost, when the model does not fit on one card anywayPaged KV with block-level eviction so a small number of very long requests do not pin the whole card ([[sched-kv-cache-aware-scheduling]])
notes
CONCURRENCY AT 128K ON A SINGLE CARD - the number an operator actually needs, which was absent from this repo. Method: reserve 12% of HBM for activations, fragmentation and CUDA graphs, subtract weights, divide the remainder by the 128K cache. Weight bytes = params_b * bytes_per_param, so a 70B in bf16 is 141.2 GB and CANNOT fit on an 80 GB H100 no matter what. RESULTS, SINGLE H100 SXM 80GB, bf16 KV then fp8 KV, concurrent 128K-token sequences: ChatGLM3-6B-128K 14.6 then 29.2 (weights only 12.8 GB). Qwen2.5-7B and Qwen2.5-7B-Instruct-1M 8.0 then 16.1 (15.2 GB). GLM-4-9B-1M 5.3 then 10.6 (18.8 GB). Llama-3.1-8B 3.5 then 6.9 (16.1 GB). Qwen3-8B 3.1 then 6.1 (16.4 GB). Yi-34B-200K 0.19 then 0.37 - a 34B with a 200K claim fits on an H100 at all only because Yi is dense, and at 128K it holds LESS THAN ONE full sequence once weights are subtracted: 69.6 GB of weights leaves 5.6 GiB against a 30 GiB sequence. DO NOT SERVE YI-34B AT 128K ON ONE H100. DOES NOT FIT ON ONE H100 EVEN AT ZERO CONTEXT: Llama-3.1-70B (141.2 GB), Qwen2.5-72B (145.4), Mistral-Large-2411 (246.0), Command-R-plus-0824 (208.0), Qwen3-235B-A22B (470.0), Qwen3-Next-80B-A3B (160.0), GLM-4.5 (710.0), Llama-3.1-405B (811.8). SINGLE H200 SXM 141GB: ChatGLM3-6B 28.0 then 56.1; Qwen2.5-7B 15.7 then 31.4; GLM-4-9B-1M 10.7 then 21.3; Llama-3.1-8B 6.8 then 13.6; Qwen3-8B 6.1 then 12.1; Yi-34B-200K 2.0 then 4.0 (69.6 GB weights on a 141 GB card finally leaves room). STILL NO FIT ON ONE H200: Llama-3.1-70B at 141.2 GB is a hair over 141 GB, Qwen2.5-72B 145.4, Llama-3.1-405B 811.8, Qwen3-235B-A22B 470.0, Qwen3-Next-80B-A3B 160.0, Mistral-Large-2411 246.0, Command-R-plus-0824 208.0, GLM-4.5 710.0. ON AN 8-GPU NODE the picture inverts, because TP shards both weights and cache: on 8xH100 80GB, minimum TP that fits weights and the resulting 128K concurrency (bf16 then fp8) is Llama-3.1-70B TP4 = 3.8 then 7.5; Qwen2.5-72B TP4 = 3.7 then 7.3; Qwen3-235B-A22B TP8 = 5.3 then 10.7; Mistral-Large-2411 TP4 = 1.2 then 2.4 (the worst number in the table: 88 layers times 8 KV heads times 128 head_dim is 352 KiB/token); Command-R-plus-0824 TP4 = 2.7 then 5.5; Qwen3-Next-80B-A3B TP4 = 44.2 then 88.4, which is the outlier good case and comes entirely from the 3-in-4 linear-attention ratio. GLM-4.5 does not fit in 8xH100 at bf16 but does at TP8 on 8xH200 (88.8 GB/card) giving 7.2 then 14.4. MLA IS A DIFFERENT REGIME ENTIRELY: DeepSeek-V3 and Kimi-K2 at TP8 put only 1.07 GiB per card per 128K sequence in bf16 (0.54 fp8, 0.27 fp4), which on an 8xH100 node with ~55 GiB of free KV per card works out to about 66 concurrent 128K sequences in bf16, 131 in fp8, 263 in fp4. That is 10x the bf16-GQA 70B number and is the single strongest argument for MLA on long-context serving. THE GENERAL LAWS, which matter more than any one row: (1) 128K CONCURRENCY IS SINGLE-DIGIT FOR MOST PRODUCTION MODELS. Across every dense-GQA 70B-and-above model in this table, one 8-GPU node holds fewer than 8 full 128K sequences in bf16 and fewer than 16 with fp8 KV. Anyone promising '128K context' without saying how many concurrent users are buying a promise that ends at the second request. (2) THE COST IS PURE MEMORY, NOT TIME - which is why it is invisible on a latency benchmark. Concurrency is a pure division; no kernel improvement changes it. (3) TP SHARDS THE CACHE, so raising TP for weight-fit reasons silently raises concurrency, and TP1-on-a-big-card vs TP8-on-small-cards is a real design choice with an 8x concurrency swing. (4) THE HEADROOM FOR QUANTIZATION IS EXACTLY 2x from bf16 to fp8 and 4x to fp4, and that is the entire reason long context is servable at all - see [[ctx-fp8-kv-cache-long-context-accuracy]] for what it costs in accuracy and [[ctx-128k-prefill-compute-cost]] for the compute side. CAVEATS: the 12% reserve is an assumption, not a measurement - vLLM's own guidance is to size for gpu_memory_utilization explicitly and to watch the KV-cache block budget, and real fragmentation on a fragmented-length workload is worse; weight bytes assume a bf16 checkpoint at the vendor's stated parameter count and ignore any quantization of weights, which would free proportionally more; no allowance is made for the logits/activation working set at 128K chunked-prefill chunk sizes; and these are steady-state ceilings, so a real deployment must hold back more for preemption and for sequences mid-admission. The MHA comparison already in the repo is 5.856 MiB/token for a DeepSeek-V3-shaped MHA model, 732 GiB at 128K, i.e. infeasible on any single card - which is what makes the 68.6 KiB/token MLA figure a 85x reduction rather than a rounding difference.

Notes

CONCURRENCY AT 128K ON A SINGLE CARD - the number an operator actually needs, which was absent from this repo. Method: reserve 12% of HBM for activations, fragmentation and CUDA graphs, subtract weights, divide the remainder by the 128K cache. Weight bytes = params_b * bytes_per_param, so a 70B in bf16 is 141.2 GB and CANNOT fit on an 80 GB H100 no matter what. RESULTS, SINGLE H100 SXM 80GB, bf16 KV then fp8 KV, concurrent 128K-token sequences: ChatGLM3-6B-128K 14.6 then 29.2 (weights only 12.8 GB). Qwen2.5-7B and Qwen2.5-7B-Instruct-1M 8.0 then 16.1 (15.2 GB). GLM-4-9B-1M 5.3 then 10.6 (18.8 GB). Llama-3.1-8B 3.5 then 6.9 (16.1 GB). Qwen3-8B 3.1 then 6.1 (16.4 GB). Yi-34B-200K 0.19 then 0.37 - a 34B with a 200K claim fits on an H100 at all only because Yi is dense, and at 128K it holds LESS THAN ONE full sequence once weights are subtracted: 69.6 GB of weights leaves 5.6 GiB against a 30 GiB sequence. DO NOT SERVE YI-34B AT 128K ON ONE H100. DOES NOT FIT ON ONE H100 EVEN AT ZERO CONTEXT: Llama-3.1-70B (141.2 GB), Qwen2.5-72B (145.4), Mistral-Large-2411 (246.0), Command-R-plus-0824 (208.0), Qwen3-235B-A22B (470.0), Qwen3-Next-80B-A3B (160.0), GLM-4.5 (710.0), Llama-3.1-405B (811.8). SINGLE H200 SXM 141GB: ChatGLM3-6B 28.0 then 56.1; Qwen2.5-7B 15.7 then 31.4; GLM-4-9B-1M 10.7 then 21.3; Llama-3.1-8B 6.8 then 13.6; Qwen3-8B 6.1 then 12.1; Yi-34B-200K 2.0 then 4.0 (69.6 GB weights on a 141 GB card finally leaves room). STILL NO FIT ON ONE H200: Llama-3.1-70B at 141.2 GB is a hair over 141 GB, Qwen2.5-72B 145.4, Llama-3.1-405B 811.8, Qwen3-235B-A22B 470.0, Qwen3-Next-80B-A3B 160.0, Mistral-Large-2411 246.0, Command-R-plus-0824 208.0, GLM-4.5 710.0. ON AN 8-GPU NODE the picture inverts, because TP shards both weights and cache: on 8xH100 80GB, minimum TP that fits weights and the resulting 128K concurrency (bf16 then fp8) is Llama-3.1-70B TP4 = 3.8 then 7.5; Qwen2.5-72B TP4 = 3.7 then 7.3; Qwen3-235B-A22B TP8 = 5.3 then 10.7; Mistral-Large-2411 TP4 = 1.2 then 2.4 (the worst number in the table: 88 layers times 8 KV heads times 128 head_dim is 352 KiB/token); Command-R-plus-0824 TP4 = 2.7 then 5.5; Qwen3-Next-80B-A3B TP4 = 44.2 then 88.4, which is the outlier good case and comes entirely from the 3-in-4 linear-attention ratio. GLM-4.5 does not fit in 8xH100 at bf16 but does at TP8 on 8xH200 (88.8 GB/card) giving 7.2 then 14.4. MLA IS A DIFFERENT REGIME ENTIRELY: DeepSeek-V3 and Kimi-K2 at TP8 put only 1.07 GiB per card per 128K sequence in bf16 (0.54 fp8, 0.27 fp4), which on an 8xH100 node with ~55 GiB of free KV per card works out to about 66 concurrent 128K sequences in bf16, 131 in fp8, 263 in fp4. That is 10x the bf16-GQA 70B number and is the single strongest argument for MLA on long-context serving. THE GENERAL LAWS, which matter more than any one row: (1) 128K CONCURRENCY IS SINGLE-DIGIT FOR MOST PRODUCTION MODELS. Across every dense-GQA 70B-and-above model in this table, one 8-GPU node holds fewer than 8 full 128K sequences in bf16 and fewer than 16 with fp8 KV. Anyone promising '128K context' without saying how many concurrent users are buying a promise that ends at the second request. (2) THE COST IS PURE MEMORY, NOT TIME - which is why it is invisible on a latency benchmark. Concurrency is a pure division; no kernel improvement changes it. (3) TP SHARDS THE CACHE, so raising TP for weight-fit reasons silently raises concurrency, and TP1-on-a-big-card vs TP8-on-small-cards is a real design choice with an 8x concurrency swing. (4) THE HEADROOM FOR QUANTIZATION IS EXACTLY 2x from bf16 to fp8 and 4x to fp4, and that is the entire reason long context is servable at all - see [[ctx-fp8-kv-cache-long-context-accuracy]] for what it costs in accuracy and [[ctx-128k-prefill-compute-cost]] for the compute side. CAVEATS: the 12% reserve is an assumption, not a measurement - vLLM's own guidance is to size for gpu_memory_utilization explicitly and to watch the KV-cache block budget, and real fragmentation on a fragmented-length workload is worse; weight bytes assume a bf16 checkpoint at the vendor's stated parameter count and ignore any quantization of weights, which would free proportionally more; no allowance is made for the logits/activation working set at 128K chunked-prefill chunk sizes; and these are steady-state ceilings, so a real deployment must hold back more for preemption and for sequences mid-admission. The MHA comparison already in the repo is 5.856 MiB/token for a DeepSeek-V3-shaped MHA model, 732 GiB at 128K, i.e. infeasible on any single card - which is what makes the 68.6 KiB/token MLA figure a 85x reduction rather than a rounding difference.

Cross-references

Hardware-dependent (3)