09 — Model memory envelope: weights, KV, and the weight-vs-KV streaming crossover

Every figure below is derived from a record in data/models/ and an accelerator record in data/accelerators/. Model dimensions were read from the records; where a record was silent or internally inconsistent, the derivation is shown from the authoritative config.json instead and marked. Nothing here is a measurement. Section 7 says what is still missing.

The central result. docs/02-flop-map.md §8 item 2 records that "No measured crossover batch for any real model on any real part." That remains true — this document does not close it. What this document closes is the analytic side, and the analytic side turns out to say something the roofline framing hides:

On H100 SXM at bf16, 61 of the 62 models here with a recorded active-parameter figure have a maximum achievable decode arithmetic intensity below the 295.4 flop/byte ridge at every context length the repo records, and are therefore memory-bound at every batch size. Six of those put the ridge out of reach inside 100 tokens of context and 37 more inside 1,000. The batch-295 crossover that docs/02-flop-map.md reports is real but it is a ceiling on a best case, not the operating point for almost any model in the repo.

1. What is computed, and from what

1.1 The ridge, per part

ridge = 1000 · dense_bf16_TFLOPs / memory_bandwidth_GB_per_s, computed from each accelerator record's own flops and memory_bandwidth_gbps fields. This is the convention already stated in docs/02-flop-map.md §3.1, recomputed here and consistent with it for H100.

docs/02-flop-map.md §8 item 1 asks for ridge points on parts other than H100 and does not have them. The accelerator records already contain every input; §3.1's table above is the answer to that open question and needs no new data. The H100 295.4 and the A100 153.0 and B200 312.5 figures match docs/02-flop-map.md §3.1 exactly, which is the cross-check.

Note the two rows that are decision-relevant and counterintuitive: NVIDIA L40S has the second-highest ridge in the repo (419) and 864 GB/s of bandwidth — a high ridge is not the same as good decode, per the wall-width warning quoted in docs/02-flop-map.md §3.1. And NVIDIA H20 has a ridge of 37, i.e. 148 dense bf16 TFLOPS against 4000 GB/s: it is memory-rich and compute-starved by design, so on this document's arithmetic it is the decode part and is a bad prefill part.

Unless a column says otherwise, every crossover in this document is H100 SXM, dense bf16, R = 295.37.

1.2 The three quantities

For each model, from its record:

(a) Weight-vs-KV streaming crossover, B_eq. Weight streaming dominates while the weights are larger than the KV read. Total bytes per decode step = W + k·L·B; total bytes attributable to weights alone = W. Weight streaming stops dominating when k·L·B > W, i.e.

B_eq = W / (k · L)

and the batch where KV streaming equals weight streaming, on an active-weight basis, is B_eq = W_act / k = 2·P·1e9 / k — a context length of one token. This is the ratio the brief asks for, and it is enormous for every model here (smallest value in the set: 17,405 for SmolLM2-1.7B-Instruct). At L = 8k divide it by 8,192. docs/02-flop-map.md §3.3's "batch 295" figure is not this number and does not conflict with it: 295 is the ridge crossing of the weight GEMM alone, while B_eq is where attention traffic overtakes weight traffic.

(b) The ridge crossing, B*. From docs/02-flop-map.md §3.3 and §3.5: the weight GEMM has AI = B, decode attention has AI = g (the GQA ratio, flat in both B and L). A decode step's combined arithmetic intensity at batch B and context L is

AI(B, L) = (2·P·B + g·k·L·B) / (W + k·L·B)

Setting AI = R and solving for B:

B* = R·W / (2P − (R − g)·k·L)

with P in units of 1e9 and k·L in bytes. Three readings follow directly:

1. g appears in the denominator, not outside it. With g omitted the formula reduces to the B* = R·W/(2P − R·k·L) that treats attention as AI = 1, which is exactly the error docs/02-flop-map.md §3.5 documents as CORRECTED. Any document computing B* for a GQA model without g is repeating a correction the repo already made.

2. B* is unbounded only while 2P > (R−g)·k·L. When that fails the denominator goes negative and there is no batch size at which the step reaches the ridge — attention traffic is already dominant and streaming it alone keeps AI below R.

3. The condition for a crossover to exist at all is a statement about context length, not batch. Define

   L_wall = 2·P·1e9 / ((R − g)·k)

   L_wall is the context length above which no batch size can reach the ridge. It is 59 tokens for SmolLM2-1.7B-Instruct and 3,577 for DeepSeek-V3 — and docs/02-flop-map.md §3.5 already states the conclusion for the general case: "NO standard attention configuration reaches the compute roof, at any context length, at any batch size, at any dtype." L_wall is that statement, per model, with a number attached.

(c) KV cache size at a context length is k · L, in GB at L = 8k / 32k / 128k / 1M, and the context at which KV exceeds 1 / 8 / 24 / 80 GB is threshold_bytes / k. All four thresholds are in the per-family tables in §3.

1.3 Field I would add to schemas/model.schema.json

I could not edit schemas/ per the brief, so recording the request here. Three additions, all backward compatible:

1. kv_cache_bytes_per_token_basis (enum: gqa | mha | mla-latent | hybrid-subset | decoder-self-attention | none-fixed-state | not-applicable). Fourteen records in this table deviate from the num_layers × num_kv_heads × head_dim × 2 × 2 formula the model schema currently documents as the derivation, and nothing in the schema marks which formula a given record used. kv_cache_bytes_per_token alone is not self-describing.
2. kv_caching_layer_count (int|null) and state_bytes_per_sequence (number|null) — for hybrid models. Nemotron-H-8B-Base-8K holds 16 KiB/token of KV plus a fixed 96 MiB per sequence of Mamba-2 state, and Falcon-H1-7B-Base holds 44 KiB/token plus 66 MiB. The fixed part is invisible to any consumer sizing memory from kv_cache_bytes_per_token alone, and it is the part that scales with concurrency.
3. attention_head_count (int|null), needed to make gqa_ratio self-checking. gqa_ratio is recorded on 62 of 64 rows here, but it is a derived quantity and two records carry a value that their own head counts do not reproduce (DeepSeek-V3 records gqa_ratio: 1.0 while num_kv_heads: 128 — correct for MLA, where the latent is expanded, but a naive num_kv_heads reader will compute g = 1 and get the wrong answer for the right reason).

2. Corrections found in existing records

Two records carry an arithmetic error in the derivation printed in their own notes, and both are in the same failure mode: the printed product does not equal the printed result. Verified against config.json on Hugging Face for each.

2.1 ‡ Phi-3-mini-4k-instruct and Phi-3.5-mini-instruct — off by 16x

Both records print:

kv_cache_bytes_per_token = num_layers * num_kv_heads * head_dim * 2 * bytes = 32 * 32 * 96 * 2 * 2 = 24576 bytes/token at bf16

32 · 32 · 96 · 2 · 2 = 393,216, not 24,576. The recorded field value (24,576) is wrong too, so this is not a typo in the prose — the number is wrong.

Correct derivation, from microsoft/Phi-3-mini-4k-instruct/config.json (fetched; num_hidden_layers 32, num_key_value_heads 32, hidden_size 3072, num_attention_heads 32, no head_dim field, so head_dim = 3072/32 = 96):

kv_cache_bytes_per_token = 32 × 32 × 96 × 2 × 2 = 393,216 B/token = 384 KiB/token

Confirmed identical for microsoft/Phi-3.5-mini-instruct/config.json — every geometry field matches, so both records take the same corrected value.

The likely origin of the error is visible in the record's own next sentence: "This is MHA, not GQA: 32 KV heads means the KV cache is 4x what a GQA-8 variant of the same width would need." 384 KiB is 4x the 96 KiB a GQA-8 model of the same width would need (32 × 8 × 96 × 2 × 2 = 98,304 B = 96 KiB). The reasoning was right; the multiplication was not carried through.

Serving impact. Phi-3.5-mini-instruct advertises 131,072 tokens of context. At the recorded 24,576 B/token that is 3.2 GB of KV per sequence; at the corrected 393,216 B/token it is 51.5 GB. The correction moves this model from "the KV is comparable to the weights" (7.64 GB weights) to "the KV is 6.7x the weights at full advertised context", and it moves the context at which KV passes 24 GB from 976,562 tokens (well past anything servable) to 61,035 tokens — inside the model's own advertised window. The 8k/32k/128k columns in §3 use the corrected figure.

This makes Phi-3 the worst KV-per-parameter dense model in the repo. Phi-3.5-mini-instruct is 3.8B parameters and carries 384 KiB/token — the same per-token KV as Olmo 3 7B, a 7.3B model, and 4x that of Llama 3.1 8B at nearly half the size. Its L_wall is 66 tokens; at any context beyond a paragraph, no batch size reaches the ridge.

2.2 GTE-Qwen2-1.5B-instruct — off by 2x

The record prints:

kv_cache_bytes_per_token = 28 * 2 * 128 * 2 * 2 = 14336 bytes/token at bf16

28 · 2 · 128 · 2 · 2 = 28,672. Verified against Alibaba-NLP/gte-Qwen2-1.5B-instruct/config.json (num_hidden_layers 28, num_key_value_heads 2, hidden_size 1536, num_attention_heads 12, so head_dim = 1536/12 = 128):

kv_cache_bytes_per_token = 28 × 2 × 128 × 2 × 2 = 28,672 B/token = 28 KiB/token

Low stakes — the record's own text correctly says the number is irrelevant to embedding use, since embedding never grows the cache — but the field value is wrong by 2x and a serving stack that sizes memory by the KV formula would be wrong.

2.3 Two derivations that are right but whose field is a worst case

Not errors, but they change what a reader should do with the number, and both records say so:

• Step-3.5-Flash records 196,608 B/token from "48 × 8 × 128 × 2 × 2 with all 48 layer slots counted." config.json gives num_hidden_layers: 45, num_attention_groups: 8, sliding_window: 512, and a layer_types array of length 48 that is 12 full_attention + 36 sliding_attention. The true steady-state cache is 12·8·128·2·2·L + 36·8·128·2·2·512, which saturates at ~688 KB per sequence regardless of length — the record states this in its own notes. The 196,608 field is a per-token figure that only applies below the 512-token window. Its L_wall of 389 tokens is therefore also wrong; the saturated model has no L_wall.
• Gemma 3 27B (multimodal) records 507,904 B/token (62 × 16 × 128 × 2 × 2) with sliding_window 1024 at a 5:1 local/global ratio. Same shape of caveat, and the record says the effective figure is roughly 10/62 of the naive one. For capacity planning the naive figure is the safe upper bound; for step-time bandwidth it is ~6x too high.

Both appear in §3 with the record value, marked here rather than in the table, because the record value is the conservative one.

2.4 Where the naive formula is right to be wrong

Confirmed correct, and worth stating because the schema's documented formula does not apply:

• MLA models. DeepSeek-V3 (70,272 = 61 × 576 × 2), DeepSeek-V2 (69,120 = 60 × 576 × 2), Kimi-K2 (70,272), GLM-5.3 (89,856 = 78 × 576 × 2), GLM-4.7-Flash (54,144 = 47 × 576 × 2). The naive GQA figure overstates by 57x (DeepSeek-V3), and docs/02-flop-map.md §2 already carries MLA's "85.3x cache reduction."
• Hybrid attention-SSM models, where only a subset of layers cache: Kimi-Linear-48B-A3B-Instruct (8,064 = 7 layers), Granite 4.0 H Tiny (8,192 = 4 layers), MiniMax-M1 (40,960 = 10 of 80 layers), Qwen3.5-35B-A3B (20,480 = 10 of 40), Qwen3.8-Flash-Next (24,576 = 12 of 48), MiniCPM-V-4_6 (12,288 = 6 of 24), Nemotron-H-8B-Base-8K (16,384 = 4 of 52), Kimi-K3 (27,648 = 24 of 93).
• Encoder-decoder. Whisper large-v3's 163,840 is decoder self-attention over decoder_layers = 32; num_layers = 32 in that record is the encoder depth, which has no KV cache at all — a bidirectional single-pass forward, nothing to cache. Whisper large-v3-turbo is the one with the 4-layer decoder (20,480). Confirmed against the safetensors/config split; the turbo record's own "FIELD-PAIR WARNING" already states this.
• Records where kv_cache_bytes_per_token: 0 is a positive statement, not a missing value. Mamba-Codestral-7B-v0.1 and RWKV-5 Eagle 7B (HF port) (fixed recurrent state, flat in context), LLaDA-8B-Instruct (diffusion, no AR cache), Parakeet CTC 1.1B and Canary-1B-Flash (encoder checkpoint) (encoder-only, no decoder). For each of these the crossover question does not arise: there is no KV stream for weight streaming to be compared against.

2.5 Independently re-derived from config.json — all agree

Ten records were recomputed from first principles against their Hugging Face config.json and every one matches the record's stored kv_cache_bytes_per_token to the byte: Llama 3.1 8B (131,072), Llama 3.3 70B (327,680), Qwen3-235B-A22B (192,512), Qwen3-30B-A3B (98,304), Qwen2.5-72B-Instruct (327,680), Mixtral 8x22B (229,376), GLM-4.5 (376,832), GPT-OSS 120B (73,728), Qwen3-8B (147,456), Olmo 3 7B (524,288), SmolLM2-1.7B-Instruct (196,608), Granite 4.0 H Tiny (81,920 naive vs 8,192 hybrid — the hybrid derivation is correct), Kimi-Linear-48B-A3B-Instruct (248,832 naive vs 8,064 hybrid — correct).

2.6 Three Gemma 4 records with kv_cache_bytes_per_token: null — now derivable

docs/02-flop-map.md and Gemma 4 31B (dense, multimodal) both record that Gemma 4's KV was left null because "the geometry is not a single formula": sliding layers use head_dim 256 while global layers use global_head_dim 512. Two geometries in one stack is not a reason to record null — it is a reason to record the sum. config.json for each of the three exposes a layer_types array that enumerates the pattern explicitly (5 sliding_attention per full_attention), plus num_key_value_heads and num_global_key_value_heads. Fetched and summed:

kv = 2 · [ (n_layers − n_global) · num_key_value_heads · head_dim + n_global · num_global_key_value_heads · global_head_dim ]

Marked * in §3. These are derivable values, not measurements, and they are marked as derived rather than asserted into the records — per the brief I have not edited data/models/. Note the result: Gemma 4 31B (dense, multimodal) at 440 KiB/token is the 6th-worst KV-per-token in the whole repo, and Gemma 4 26B A4B (MoE, multimodal) at 110 KiB/token is 3.2x cheaper than its 25.81B total suggests, because a 4B-active MoE with a 2-head global geometry caches very little per token.

2.7 Records that cannot be computed, and why

Recorded as no record rather than estimated:

• Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 4.5, GPT-4.1, o3 — architecture undisclosed. Every structural field is null by design and the records say so explicitly. Anthropic and OpenAI publish no layer count, head count, or attention type, so no KV-per-token figure is derivable and inventing one would be the exact failure mode this repo exists to prevent.
• DeepSeek-V4-Pro, DeepSeek-V4-Flash — kv_lora_rank is absent from both configs and the compressed-cache geometry (compress_ratios, sliding_window 128, hc_mult) is not resolvable from the fields present. The records warn against carrying the V3 latent formula forward, which is correct.
• Parakeet TDT 0.6B v2, PaliGemma 3B mix 224 — no machine-readable per-layer dimensions available; both records already flag incomplete verification.
• Grok 1, GLM-5.3 — kv_cache_bytes_per_token and gqa_ratio are present, but active_params_b is null, so W_act, B_eq and L_wall cannot be computed. GLM-5.3's own notes decline to state an activated figure, and estimate ~40.9B from config as "indicative only." I have carried that discipline: no record, not 40.9.

3. Per-family tables

All KV figures bf16. active weights GB = 2 · active_params_b · 1e9 / 1e9. B_eq and L_wall are as defined in §1.2. * = KV derived in §2.6, not in the record. ‡ = corrected in §2.1.

A note on reading active weights GB: for a MoE this is not the memory the card must hold. GPT-OSS 120B shows 10.20 GB active but 116.8B total = 233.6 GB of bf16 weights that all have to be resident — the record's own note says "all 117B must be resident even though only 5.1B is active," and explains that this is exactly why the released MXFP4 build fits on one 80GB device. The B_eq (total basis) column in §4 carries the resident-weights version of the crossover.

Dense — 34 models

MoE — 20 models

Hybrid attention-SSM — 10 models

4. The crossover, and what it is actually for

B_eq on an active-weights basis is a ratio of two numbers in the record and needs no accelerator at all:

B_eq = 2 · active_params_b · 1e9 / kv_cache_bytes_per_token

This is the number that says how much context you need before KV streaming matters at all. Its physical reading: at batch B_eq, one step streams as many KV bytes as it streams weight bytes, on a single-token-per-sequence basis. Multiply by L for the real batch requirement at context L.

4.1 B_eq ranked — smallest first (weight streaming stops dominating soonest)

Read this ranking carefully — the brief's framing inverts it. The question asked was which models "stay memory-bound longest, so care most about bandwidth not FLOPS." The lowest-B_eq models are the ones where KV streaming overtakes weight streaming soonest, which means they are the models where bandwidth matters most and weight-streaming optimization matters least — but not the models most stuck on the memory wall. B_eq is a crossover between two memory streams; both sides are memory traffic. The models that are stuck on the memory wall are the ones with a low L_wall, which is a different and much more severe condition (§4.2).

The two are nonetheless mechanically linked, and the reason is visible in the top rows: a low B_eq requires either small P or large k, and large k at small P is precisely what pushes L_wall down. SmolLM2-1.7B-Instruct tops the B_eq table and has the lowest L_wall in the repo (59 tokens). The first eleven rows of this table and the first eleven rows of §4.2 are largely the same models.

4.2 L_wall ranked — the models that cannot reach the compute roof at any batch

L_wall = 2·P·1e9 / ((295.37 − g)·k). Above this context length, no batch size makes a decode step compute-bound on H100 bf16.

The "AI ceiling" columns are AI(B→∞, L) = 2P/(k·L) + g — the highest arithmetic intensity achievable at that context length at any batch size. Compare to R = 295.4. Every value in the table is 6x to 50x below the ridge. Gemma 3 27B (multimodal) at 128k context has an AI ceiling of 2.8 flop/byte against a ridge of 295.4 — 106x below it, at infinite batch. This is docs/02-flop-map.md §3.5's "NO standard attention configuration reaches the compute roof" as a per-model table.

Three of the twenty are also the worst models in the repo on raw KV bytes (§5): Gemma 1 7B (448 KiB/token at 8.5B parameters), Gemma 3 12B IT (multimodal) (384 KiB), Gemma 3 27B (multimodal) (496 KiB). The Gemma family is consistently the worst KV geometry in this repo, and it is worst because of head_dim: 256 with few KV heads — Gemma 2 and 3 use head dimensions of 256 where Llama and Qwen use 128, doubling per-head KV for the same hidden size. That is the mechanism, and it is in the records (head_dim: 256 on every Gemma 2/3 record, head_dim: 128 on every Llama/Qwen 3).

4.3 B_eq on a resident-weights basis — where MoE sparsity changes the answer

For a MoE, the weights a card must hold are the total, not the active set. Recomputing B_eq with W = 2 · params_b · 1e9:

This is the MoE crossover penalty stated as a ratio, and it is the reason the MoE expert GEMMs (the sparse feed-forward compute) record's AI = B·(k/E) matters at the memory level and not only the FLOP level. Kimi-K2 streams 2.05 TB of weights to do 32B active; on the active basis the crossover looks like batch 911k, on the resident basis it is 29.2 million. There is no serving configuration that reaches the latter.

The exception that proves the mechanism: GLM-4.7-Flash has the smallest MoE active-to-total ratio in the table (31.2B total / 3B active = 10.4x) and therefore the smallest crossover penalty — and it got there by adopting MLA, which took its KV from a hypothetical 368 KiB/token down to 52.9. GLM-4.5 and GLM-4.7-Flash are the cleanest before/after pair in the repo: same vendor, same 368-vs-53 KiB question, 7x KV reduction against a 32B-active model.

4.4 The B* numbers that exist, and why almost none do

Evaluating B* = R·W / (2P − (R−g)·k·L) on the active-weights basis:

† computed on a total-weights basis because active_params_b is null in both records (§7.4); the other rows are active-weights basis.

"none" = the denominator 2P − (R−g)·k·L is ≤ 0, i.e. no batch size reaches the ridge at that context length. 19 of the 64 rows have a B* at 1k tokens of context, and only 2 of those survive to 8k (Kimi-K3 at 434, Grok 1 at 28,178). Exactly one survives to 32k — GLM-5.3 at batch 695, on a 753B model whose weights cannot fit on the card the ridge was computed for. The other 45 rows have no B* at any context length the repo records. active_params_b is non-null for the other 62 rows.

Where B* does exist at 1k it is not far above the 295 that docs/02-flop-map.md quotes: DeepSeek-V3 at 413, Kimi-K3 at 307, GLM-5.3 at 300. That is a useful sanity result — the roofline figure is approximately right at short context, and the whole model-specific contribution is showing how quickly it stops being applicable. The six widest models in the table (Hermes-4-405B 361, MiniMax-M1 340, Grok 1 337, Llama 3.3 70B 932, Qwen2.5-72B-Instruct 876) get closer to 295 as they get wider, because a larger P pushes the attention term down relative to the weight term — which is the same mechanism docs/02-flop-map.md §5 describes as "a WIDER model defers the attention crossover further out," now expressed as a batch instead of a context length.

4.5 What this means for the gap in docs/02-flop-map.md §8.2

The gap says no measured crossover exists and that Dense weight GEMMs in the decode step (the per-token matmul tax) warns real kernels cross earlier than the roofline. The per-model picture sharpens what is being asked for: a single "measured crossover batch" is not a well-defined quantity for these models, because for 47 of 64 there is no analytic crossover to be early relative to. What is measurable, and what the repo lacks, is:

1. Achieved-vs-peak bandwidth at batch 1 per model — bounded above by the AI ceiling column in §4.2. For Gemma 3 27B (multimodal) the ceiling is 2.8 flop/byte, so a measured step time should be within ~3x of bytes/3.35 TB/s no matter what batch size is used, and any throughput gain from batching must come from amortizing bandwidth (fewer weight passes), never from reaching compute.
2. The batch at which aggregate decode throughput stops scaling — which for these models is set by memory capacity and kernel efficiency, not by the ridge. The only existing measurement of this shape in the repo is Per-user decode tok/s, MPT-7B static batch 1, 1x A100 against Aggregate output throughput, MPT-7B static batch 64, 1x A100: batch 1 → 64 raises aggregate throughput 13.9x (57.6 → 800 tok/s) while per-user decode speed falls 4.6x (57.6 → 12.5 tok/s). That is a 13.9x throughput gain over a 64x batch increase — i.e. efficiency per token fell 4.6x, which is the empirical signature of staying memory-bound the whole way. MPT-7B is a 7B model whose B_eq is ~89,000 and whose L_wall is well under 1,000 tokens, so this is exactly the predicted regime and the prediction holds. But it is one model, one engine (FasterTransformer, static batching), one part, and a 40GB A100 the repo has no accelerator record for — the benchmarks say so themselves. It is evidence, not a measurement of the crossover.

I have not added a benchmark record for it, because both numbers already exist as records and a third would be a duplicate, not a measurement.

5. The 10 models with the worst long-context memory envelope

Ranked by the context length at which KV cache for one sequence passes 80 GB — beyond this a single request cannot fit on an 80GB card, whatever the weights do. 80GB is NVIDIA H100 SXM's vram_gb.

Four observations that change how this table is used:

1. Only one of the ten — Gemma 4 31B (dense, multimodal)*, at 118.1 GB — actually breaks an 80GB card at its own advertised context. Two more come close enough to matter operationally: Hermes-4-405B (67.6 GB) and Gemma 3 27B (multimodal) (66.6 GB) each advertise 128k and would leave ~12 GB for weights, activations and workspace — and for Hermes-4-405B the weights alone are 811.71 GB in bf16, so it needs quantization and sharding regardless. The table ranks KV geometry, not usability: rows 1, 2, 5 and 9 advertise contexts far below their own 80GB threshold and are comfortably servable.
2. GLM-4.5 is the only MoE in the top ten and the only model in it that could have been fixed. Its 368 KiB/token is the direct consequence of plain GQA, and the record says so explicitly: "GLM-4.5 does 32B active in 368 KiB/token — 5.4x the KV for slightly less compute" versus DeepSeek-V3's 68.6 KiB. Its own successor GLM-4.7-Flash fixed it with MLA at 52.9 KiB. The fix is one architectural decision, already made by the same vendor one generation later.
3. The Gemma pattern is head_dim: 256. Every Gemma 2/3 model in the top ten has 256-wide heads against Llama's and Qwen's 128, and Gemma 4 31B (dense, multimodal)* stacks that on top of 60 layers. This is a design choice with a long history, not a bug, but it is the single most consequential memory decision across the models in this repo.
4. Phi-3.5-mini-instruct would not have appeared here at all with its recorded (erroneous) 24,576 B/token — that value puts it 56th of 64 in this ranking. The §2.1 correction moves it to 8th. The comparison that makes the correction matter: at 3.82B parameters it carries exactly the same 384 KiB/token as Olmo 3 7B, a 7.3B model, and 3x the KV of Phi-4-mini-instruct at 3.84B — a near-identical sibling one release later that added GQA (8 KV heads instead of 32). Only 4 of the 64 rows in §3 carry more KV per token.

5.1 Models whose advertised full context exceeds 80 GB of KV for one sequence

Six, on the raw max_position_embeddings figure. Three of the six are excluded below because the record itself says the advertised context is not a serving figure:

The two excluded rows are the same failure mode in both cases, and both records warn about it in their own notes:

• Llama 4 Scout (17B active / 109B total) carries the same 10,485,760 max_position_embeddings as Maverick. At 192 KiB/token that is 2,061.6 GB of KV. Its own record calls the figure "a nominal training/rope-range figure, NOT a usable serving context" and notes real deployments land at 10M only with heavy KV offload. Not reported as a finding.
• MiniMax-M1's 10,240,000 gives 419.4 GB naively, but that over-counts: the record's 40,960 B/token is already net of the hybrid pattern (10 full-attention layers of 80), so the figure is arithmetically correct and genuinely unusable — the record notes the context was "built for very long RL/test-time-compute runs," not for KV-resident serving.

6. The cheap end, for contrast

The bottom of the table is where the architectural decisions pay, and the spread is two orders of magnitude in KV per token across models of similar size:

The three structural levers, ranked by effect, all visible in the table:

1. Hybrid attention (Kimi-K3: 27 KiB/token at 2.78T total / 104B active). 7.5M total parameters and the lowest-but-few KV in the repo. Its record states the ratio directly: "KV per token is LOWER than Qwen3-8B's 144 KiB (a dense 8B) and less than half of DeepSeek-V3's 68.6 KiB."
2. MLA (DeepSeek-V3: 68.6 KiB/token for 37B active, against GLM-4.5's 368 KiB for 32B active — 5.4x less KV for more compute). Confirmed by Multi-head Latent Attention: KV cache compressed to a single low-rank latent vector per token's 85.3x reduction against the naive GQA figure.
3. Aggressive GQA (GPT-OSS 120B: 72 KiB/token at g=8, on a 116.8B MoE that fits one 80GB device only because its weights ship MXFP4).

And the counter-example in the same table: Olmo 3 7B is a 7.3B model with 512 KiB/token — the worst in the repo — because it is MHA (32 query / 32 KV heads, g=1). A model small enough to be irrelevant on every other axis is undone entirely by KV geometry. That is the clearest possible statement of what kv_cache_bytes_per_token is for.

One caution on the hybrid rows. These models' small KV figures come with a fixed per-sequence state that the table above does not show: Nemotron-H-8B-Base-8K holds 96 MiB per sequence of Mamba-2 state (48 layers × 2 MiB), Falcon-H1-7B-Base holds 66 MiB, Mamba-Codestral-7B-v0.1 holds 128 MiB. Both numbers are in those records' notes. The fixed part scales with concurrency rather than context, so at 100 concurrent sequences it is 9.4 GB for Nemotron-H — larger than its entire 128k KV budget. kv_cache_bytes_per_token alone is an incomplete memory model for every hybrid model in this repo, which is why §1.3 asks for state_bytes_per_sequence.

7. What this document does not establish

1. No measurement. Every crossover here is analytic, exactly as docs/02-flop-map.md §8.2 says. Dense weight GEMMs in the decode step (the per-token matmul tax)'s warning stands unchanged: "Real kernels hit their crossover earlier than the ideal roofline." §4.4 shows that for 47 of 64 models there is no analytic crossover to be early relative to, which makes that warning harder to apply, not easier.
2. The ridge is anchored to H100 SXM bf16. §1.1 gives the per-part ridges but the crossover arithmetic is only worked for H100. On NVIDIA A100 80GB SXM4 (R = 153.0) the L_wall values roughly double; on NVIDIA L40S (R = 419.0) they roughly halve. No model in this table has an L_wall above ~57,000 tokens, so no part in the repo's accelerator set makes a single one of these models compute-bound at a realistic serving context. That is a stronger and more uncomfortable version of the claim in §8.
3. The AI expression assumes the ideal weight-GEMM AI = B. A real MoE decode step has AI = B·(k/E) per MoE expert GEMMs (the sparse feed-forward compute) — so for the 20 MoE rows, the true AI is lower than computed here and the true L_wall is correspondingly higher. DeepSeek-V3's k/E = 1/32 means the real AI ceiling at 8k is not 129.5 but ~5, and its real L_wall is not 3,577 but ~115,000. The MoE rows in §4.2 are therefore optimistic, and the correct reading is that MoE models are more memory-bound than the table shows.
4. active_params_b is null on 2 records — Grok 1 and GLM-5.3 — so W_act, B_eq and L_wall are no record for them and only the total-weights B* is given (§4.4). GLM-5.3's own note declines to state an activated figure and estimates ~40.9B from config as "indicative only"; that estimate is not used here. No model dimension was invented anywhere in this document — where a record is silent, the table says no record.
5. The * Gemma 4 rows are derivations, not records. §2.6 gives the arithmetic and the config fields. They are marked everywhere they appear.
6. fp8/int8 KV is not modelled in §3–§5, but it can be bounded. Per Attention over the KV cache in the decode step, halving KV bytes doubles attention AI to 2g, so with k' = k/2 and g' = 2g the wall moves to L_wall' = 2P/((R−2g)·k/2) — L_wall grows by 2(R−g)/(R−2g), which is 2.007x at g = 1 and 2.122x at g = 16. That lifts DeepSeek-V3 from L_wall 3,577 to 7,179 tokens and SmolLM2-1.7B-Instruct from 59 to 119 — real, and still far below any serving context. So on H100 bf16 the KV-dtype lever is a capacity lever, not a compute-bound lever: it halves every 80GB threshold in §5 and changes no conclusion about the roofline. Same conclusion the Attention over the KV cache in the decode step record reaches by a different route.

Related documents

• 02-flop-map.md — the ridge points, the flop classes, and §8's list of what the records cannot answer. This document is the per-model completion of its §3.1 table and its §8.2 gap.
• 01-hardware-selection.md — which part to buy for each regime; §1.1 here gives the ridge input that selection needs.
• 04-quant-selection.md — the one lever that changes the arithmetic intensity without touching the model, and the reason most of the large models in §5 are quantized in practice.
• 05-known-traps.md — the software failures behind §4's crossovers not materialising.