Attention over the KV cache in the decode step

Structurally ~1 flop/byte for MHA at EVERY context length and EVERY batch size - and CONFIRMED by KIVI's abstract, which states the two-part failure independently: as batch size and context length grow the KV cache 'becomes the new bottleneck in speed and memory usage', AND separately 'the loading of the KV cache causes the computational core to be idle, which limits the inference speed'. An idle compute core is the arithmetic-intensity statement in plain words. Derivation, per decoded token, one query row, n_q query heads, n_kv KV heads, head_dim d_h, context length L, bf16 KV: flops = 2*L*d_h*n_q (QK^T) + 2*L*d_h*n_q (PV) = 4*L*d_h*n_q; bytes = 2 tensors (K and V) * L*d_h*n_kv * 2 bytes = 4*L*d_h*n_kv. AI = 4*L*d_h*n_q / (4*L*d_h*n_kv) = n_q/n_kv = g, the GQA group ratio, independent of B AND independent of L. CORRECTION to an earlier version of this field: it stated AI = 1 flop/byte unconditionally. That is true only for MHA (g=1). For Llama-3-8B (32 query / 8 KV heads) AI = 4; for Llama-2-70B (64 query / 8 KV heads) AI = 8; for MQA with 32 query heads AI = 32. The qualitative conclusion survives and is if anything strengthened - AI is a constant of the architecture, not a function of any serving knob - but the magnitude was wrong by the group ratio, and it matters because that ratio is exactly what GQA/MQA and KV-cache dtype quantization are being asked to buy. With int8/fp8 KV (1 byte per element instead of 2) bytes halve and AI doubles, so AI = 2g: Llama-3-8B goes from 4.0 to 8.0 flop/byte. Against H100 SXM's dense-bf16 ridge of 295.4 flop/byte (989.5/3350), the gap is: MHA bf16 KV 295x below; GQA-8 bf16 37x below; GQA-8 int8 18x below; and even the aggressive combination MQA with 64 query heads (g=64) at int8 KV gives AI = 128, still 2.3x below the ridge. So NO standard attention configuration reaches the compute roof, at any context length, at any batch size, at any dtype. The consequence is sharper than for prefill: because flops and bytes both scale linearly in context length, growing context does NOT raise arithmetic intensity, and batching does NOT raise it either. Continuous batching therefore gives no relief here, unlike for decode_gemm, and long-context ITL degrades linearly in L for every sequence in the batch at once - there is no batch size at which the wall recedes.

This is the class that makes long-context serving expensive, and the reason is structural rather than a kernel deficiency: a decode step reads the entire KV cache to produce one token, and no amount of batching or tiling changes the bytes-per-flop ratio. The classification matters for engine authors because it rules out the fix that works everywhere else. For decode_gemm, raising batch size moves the workload across the roofline; for decode_attention, batch size does not appear in the AI expression at all, so continuous batching - the standard cure for memory-bound decode - does nothing here. What DOES work, in order of how much AI each buys: int8/fp8 KV (x2), GQA/MQA at the model's own group ratio (xg, but a model-selection decision rather than a serving decision), and then the structural family - MLA, sliding window, sparse/hardware-aligned attention (NSA), or replacing attention layers with SSMs entirely - which is the only group that changes WHICH bytes are read rather than how many bits each costs. Hardware implication: parts with high memory bandwidth and a large pool (HBM, and unified/Apple-style memory) win long-context decode; small-VRAM consumer parts degrade roughly linearly in context length once the KV cache no longer fits and paging begins. Engine implication: measure long-context inter-token latency separately from short-context latency - a part that looks fine at 2k can be an order of magnitude off at 32k, and continuous batching will not save it because AI is batch-invariant. Engineering corollary, and the one place better math still pays: since flops here scale linearly with context rather than being negligible as in the weight-GEMM case, fp8 attention and sparsity genuinely reduce time on this class, whereas extra FLOPS on the GEMMs would not. Quantify with KV bytes per token = 2 (K and V) * n_layers * n_kv_heads * head_dim * bytes_per_element: Llama-2-7B (MHA, 32 layers, 32 KV heads, 128 head_dim, bf16) = 2*32*32*128*2 = 512 KiB/token, so 16 GiB of KV per sequence at 32k context; Llama-2-70B (GQA-8, 80 layers, 8 KV heads) = 2*80*8*128*2 = 320 KiB/token, 10 GiB at 32k; Llama-3-8B (GQA-4, 32 layers, 8 KV heads) = 2*32*8*128*2 = 128 KiB/token, 4 GiB at 32k, halving to 2 GiB at int8. The GQA ratio is a 4-8x reduction in exactly the quantity that bounds this class, which is why a GQA model degrades far more gracefully in ITL than an MHA model of similar size - and it does so without changing a single flop. Note the trap in the head counts: model-card prose for Llama-2-70B describes 64 KV heads while its config.json specifies 8, and using 64 would overstate KV bytes by 8x. Take n_kv from the config. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.