kv_cache_bytes_per_token = 0 is three different answers wearing one number, and a capacity plan built on it is wrong in two of the three cases

A reader sums kv_cache_bytes_per_token across a fleet to size KV memory and gets a plausible-looking number, because the field is populated on every model record and is a number everywhere. The field carries THREE semantically different answers behind the single value 0, and the two that are not 'there is genuinely no KV' are the ones that break a capacity plan. Counted across data/models/ on 2026-10-04: 35 records carry 0, 14 carry null. The 35 split into three groups. (A) TRULY ZERO - the model holds no KV cache because it does one bidirectional forward pass with no autoregressive streaming: every BERT-family embedder and reranker in the set (all-minilm-l6-v2, bge-*, e5-*, gte-*, cohere-*, jina-*, mxbai-*, nomic-*, voyage-*, snowflake-arctic-embed-l, modernbert-base), the FastConformer speech encoders (canary-1b-flash, parakeet-ctc-1-1b), and the pure recurrent/diffusion models with no attention layers at all (mamba-codestral-7b 'pure Mamba-2, no attention layers at all', rwkv5-eagle-7b 'linear-attention recurrent (WKV) mixer', llada-8b-instruct and mercury as diffusion). For these 0 is the whole truth and planning on it is correct. (B) ZERO WITH A NONZERO ATTENTION SLICE - LFM2. lfm2-1-2b and lfm2-5-1-2b-instruct are recorded 0 and their own notes state why: 'the repo's own convention for state-space/hybrid models is that there is no growing KV cache to plan for', and both notes ALSO carry the number that matters, 6 attention layers x 8 kv heads x 64 head_dim x 2 x 2 = 12,288 bytes/token, which at the model's advertised 128000 max_position_embeddings is 1.5 GiB. So the record says 0 and means 'not O(L), but not nothing'. (C) ZERO WHERE THE MODEL ARCHITECTURE MAKES THE NAIVE FORMULA OVER-PROVISION BY 50x OR MORE - this is the group that actually hurts. deepseek-v3, deepseek-v2, deepseek-v3-2, deepseek-r1 and kimi-k2 all carry 70,272 or 69,120 bytes/token, against a naive num_layers x num_kv_heads x head_dim x 2 x 2 of 3,997,696 and 3,932,160 respectively - MLA's kv_lora_rank 512 latent compresses the cache by about 57x, so a planner who computes KV from geometry and compares against 0 concludes the model has no cache when in fact it has one 57x SMALLER than the formula predicts.

One numeric field was made to carry three different claims: 'this architecture has no KV cache', 'this architecture has a KV cache that does not grow with context', and 'this architecture's KV cache is compressed by a mechanism that defeats the standard formula'. The first is a fact about the architecture, the second is a fact about scaling, and the third is a fact about a specific attention mechanism. A single 0-valued field cannot distinguish them, and the distinction determines whether the correct planning action is 'allocate nothing', 'allocate a bounded amount independent of context', or 'allocate an amount 57x smaller than your formula says'.

Never branch on the numeric value of kv_cache_bytes_per_token alone. Branch on model architecture and attention_variant, which are populated on every record and which DO distinguish the three cases: architecture 'dense' with a bidirectional-encoder or cross-encoder attention_variant is case A (genuinely nothing); architecture 'hybrid-attention-ssm' or 'ssm' or 'recurrent' is case B (bounded, not growing - read the attention-layer count out of attention_variant and multiply it out, the LFM2 records show the worked example); and attention_variant naming MLA or a latent cache is case C (apply kv_lora_rank, not num_kv_heads x head_dim). Treat null as a fourth, distinct answer meaning 'not disclosed', which is what the 14 null records mean and which must never be treated as 0 - gpt-4-1, the claude family, deepseek-v4-flash/pro, o3, griffin-14b, jamba-1-5-large, samba-7b, paligemma-3b-mix-224, parakeet-tdt-0-6b-v2, retnet-arch and hyena-arch are unknown, not zero.

Found by recomputation, not by reading: a 2026-10-04 cross-check recomputed kv_cache_bytes_per_token from model geometry for all 155 model records that carry it. 112 were checkable from num_layers x num_kv_heads x head_dim alone; 32 of those departed from the naive formula and EVERY ONE OF THE 32 was explicable and correctly documented - 5 MLA models verified EXACTLY against 61*(512+64)*2 = 70,272 and its siblings, 8 hybrid-attention models verified EXACTLY against (attention-layer count) x num_kv_heads x head_dim x 2 x 2, and the Whisper family verified against DECODER layers only. The two LFM2 records initially looked like errors for exactly the reason this gotcha describes - a mechanical check flags 'recorded 0, expected 12,288' - and on reading them they turned out to be deliberate and documented. So this is NOT a data error to fix; it is a schema limitation, and the honest fix is a discriminator field (for example kv_cache_growth: none | bounded_in_context | context_linear) rather than a change to any recorded number. LFM2's 0 was NOT changed, deliberately: the records' reasoning is sound and consistent, and overwriting a deliberate, explained convention would destroy the explanation. Related: [[hybrid-attn-ssm-kv-cache-sizing]], [[attn-sliding-window-ssm-fixed-state-cost]], [[mla-latent-attention]], [[ctx-kv-cache-gib-at-claimed-context]], [[ssm-decode-cost-o1-vs-attention-on]]. The encoder-side roofline inversion that makes case A correct in the first place: [[eng-profile-encoder-serving-inverts-the-roofline]].