inference/research
verifiedhigh · 0.85updated 2026-10-05

Prefill FLOP crossover: the context length at which attention equals the MLP, computed per architecture family from each model's own config

compute bound

Fields

id
flops/fct-prefill-attention-gemm-crossover-by-family
Class
prefill_attention
Bound by
compute
Arithmetic intensity
THE CROSSOVER IS A CONFIG RATIO, NOT A CONTEXT NUMBER. Within one prefill of L tokens, per layer: attention flops = 2*n_q*head_dim*L^2 (causal: the two L x L score/output matmuls each contribute 2*n_q*head_dim*L^2 before halving, and the mask halves both, so the factor 2 already includes it); GEMM flops = 2*P_layer*L. Setting them equal gives L* = P_layer / (n_q*head_dim) - a pure parameter ratio with NO hardware in it and no context in it. And the true share of prefill flops is not L/L* but A(L) = L/(L + L*), because the two terms add rather than compete; A(L*) = 50% by construction. Where n_q*head_dim = hidden_size (the normal case), L* = P_layer/hidden_size, which is roughly the feed-forward expansion ratio plus the attention projections - so L* is a number in the tens of thousands for a dense transformer and the crossover is NEVER reached inside a normal context window. THE TABLE, every geometry read from the model's own config.json (head counts, head_dim, intermediate_size), P_layer computed as q: d*n_q*d_h, k and v: 2*d*n_kv*d_h, o: n_q*d_h*d, mlp: 3*d*I (gate, up, down): Llama-2-7B (MHA, d=4096, I=11008, n_q=32, n_kv=32, d_h=128) P_layer = 202,375,168, n_q*d_h = 4096, L* = 49,408; Llama-2-13B (MHA, d=5120, I=13824, 40 heads, 40 KV heads) P_layer = 317,194,240, n_q*d_h = 5120, L* = 61,952; Llama-2-70B (GQA-8, d=8192, I=28672, n_q=64, n_kv=8) P_layer = 855,638,016, n_q*d_h = 8192, L* = 104,448; Llama-3-8B (GQA-4, d=4096, I=14336, 32/8 heads) P_layer = 218,103,808, L* = 53,248; Mistral-7B-v0.1 (GQA-4, d=4096, I=14336, 32/8) P_layer = 218,103,808, L* = 53,248 - IDENTICAL to Llama-3-8B because the two configs are geometrically identical, which is a real property of those two models and not a transcription error; Mixtral-8x7B (top-2 of 8, d=4096, I_expert=14336, 32/8) P_layer = 394,264,576, L* = 96,256 - 1.8x Llama-3-8B because the dense Llama layer activates 3*4096*14336 of MLP weight while Mixtral's top-2 activates 3*4096*14336 too but the expert count changes routing, not active width, so the increase is from counting the expert FFN at full per-expert width while the Llama number counts one FFN; Qwen3-32B (dense, d=5120, I=25600, n_q=64, n_kv=8, d_h=128) P_layer = 487,587,840, n_q*d_h = 8192, L* = 59,520; Qwen3-30B-A3B (top-8 of 128, d=2048, I_expert=768, moe_intermediate=768, 32/4 heads, d_h=128) P_layer = 56,623,104, n_q*d_h = 4096, L* = 13,824 - and THIS is the row that matters, because it is 4x lower than any dense model in the table. ATTENTION SHARE A(L) = L/(L+L*) at 1k / 8k / 32k / 128k: Llama-2-7B 2.0% / 14.2% / 39.9% / 72.6%; Llama-2-13B 1.6% / 11.7% / 34.6% / 67.9%; Llama-2-70B 1.0% / 7.3% / 23.9% / 55.7%; Llama-3-8B 1.9% / 13.3% / 38.1% / 71.1%; Mistral-7B 1.9% / 13.3% / 38.1% / 71.1%; Mixtral-8x7B 1.1% / 7.8% / 25.4% / 57.7%; Qwen3-32B 1.7% / 12.1% / 35.5% / 68.8%; Qwen3-30B-A3B 6.9% / 37.2% / 70.3% / 90.5%. TWO FAMILIES BREAK THE FORM, and both are why a single number is wrong. (1) HYBRID attention-SSM: only the attention layers have an L^2 term, so divide L* by the attention-layer fraction. Nemotron-H-8B (52 layers, hybrid_override_pattern gives exactly 4 attention layers, d=4096, I=21504, 32/8 heads, d_h=128): dense-layer P_layer = 306,184,192 so L* = 74,752 for a full-attention layer, but 4/52 = 0.0769 of layers are attention, so the effective crossover is L*_eff = 5,750 and A(L) = 15.1% at 1k, 58.8% at 8k, 85.1% at 32k - a hybrid model is an attention-dominated model at 8k where a dense Llama of the same size is at 13%. Gemma-3-27B (62 layers, sliding_window 1024 with sliding_window_pattern 6, so 1 layer in 6 attends globally and 5 attend only to the last 1024 tokens, d=5376, I=21504, 32/16 heads, d_h=128): P_layer = 412,876,800, L* = 100,800 for a global layer, and since only the 1-in-6 global layers have an L^2 term the effective crossover is 16,800, giving 5.7% / 32.8% / 66.1%. The correct treatment for Gemma-3 is not a single L* at all: attention flops are A_global(1/6)*L^2/2 + A_local(5/6)*min(L,1024)^2/2, so beyond L = 1024 the local layers stop growing and only the global layer's L^2 term remains - which is why a sliding-window model has an attention share that PLATEAUS instead of rising toward 1. (2) MLA (latent attention): DeepSeek-V3 compresses K and V into a 512-rank latent, so per-layer attention flops = 2*n_q*(kv_lora_rank + qk_rope_head_dim)*L^2 for QK plus 2*n_q*v_head_dim*L^2 for PV, i.e. the denominator n_q*head_dim is replaced by n_q*(192 + 128) = 40,960 in the QK half and n_q*128 = 16,384 in the PV half - an effective ~28,672 against P_layer = 866,123,776 (top-8 of 256 routed experts at moe_intermediate 2048 plus one shared expert), giving L* ~ 52,864 on the PV half alone and ~30,205 on the QK half, so MLA's crossover is in the same range as a dense model's while its KV BYTES are 85x smaller. Two consequences a reader should carry away. FIRST, the crossover is not a hardware property at all: it is P_layer/(n_q*head_dim), computed from a config file, identical on every part. Hardware decides only WHICH side of the crossover a given prefill falls on, via the ridge crossings L_ridge = 2*AI_ridge/g from prefill-attention (591 tokens for MHA on H100, 148 for GQA-4) - which are hundreds of tokens, not tens of thousands. So in every configuration in this table, prefill is already compute-bound long before the attention share is significant, and the two numbers answer different questions. SECOND, the direction of the comparison surprises people: a WIDER model defers the crossover (Llama-2-70B at 104,448 vs Llama-2-13B at 61,952) while a MOE model with a narrow hidden size advances it enormously (Qwen3-30B-A3B at 13,824), because L* is a ratio of per-token GEMM width to attention width and MoE deliberately shrinks the numerator. A 30B MoE is therefore far more attention-bound than a 32B dense model, which is a fact about the config and not about the parameter count.
Scales with
context_length (attention share = L/(L+L*), rising monotonically toward 1 for dense attention)intermediate_size I (numerator of L* via 3*d*I - the single biggest lever)num_attention_heads x head_dim (denominator of L* - note these need not equal hidden_size; Qwen3 sets head_dim independently)attention-layer fraction (the divisor for hybrid attention-SSM models: Nemotron-H 4/52, Gemma-3 1/6 global)sliding_window (caps the local layers' L^2 term, which makes the attention share plateau instead of rising)active_experts_per_token (numerator of L* for MoE - top-8 of 128 gives a far smaller numerator than a dense FFN of the same model size)kv_lora_rank (denominator for MLA's QK half)
Hardware parameters
parametermemory_bandwidtheffectWidening the FFN per layer pushes L* out proportionally - a design lever, not an inference one, but it is why 70B-class models are less attention-heavy at 32k than 7B-class ones | Sliding-window attention bounds the L^2 term and is the only mechanism in this table that makes the attention share PLATEAU rather than approach 1 (Gemma-3: local layers stop at 1024 tokens) | Chunked prefill (SARATHI, arXiv 2308.16369) ...basisunverifiedvalues_fornvidia-h100-sxm,amd-instinct-mi300x,nvidia-b200,google-tpu-v5eparametertensor_core_counteffectWidening the FFN per layer pushes L* out proportionally - a design lever, not an inference one, but it is why 70B-class models are less attention-heavy at 32k than 7B-class ones | Sliding-window attention bounds the L^2 term and is the only mechanism in this table that makes the attention share PLATEAU rather than approach 1 (Gemma-3: local layers stop at 1024 tokens) | Chunked prefill (SARATHI, arXiv 2308.16369) ...basisunverifiedvalues_fornvidia-h100-sxm,amd-instinct-mi300x,nvidia-b200,google-tpu-v5e
Workarounds
Widening the FFN per layer pushes L* out proportionally - a design lever, not an inference one, but it is why 70B-class models are less attention-heavy at 32k than 7B-class onesSliding-window attention bounds the L^2 term and is the only mechanism in this table that makes the attention share PLATEAU rather than approach 1 (Gemma-3: local layers stop at 1024 tokens)Chunked prefill (SARATHI, arXiv 2308.16369) bounds the attention tile size so a long prompt does not allocate a full L x L score matrix, trading peak bandwidth for bounded memoryFlashAttention / FA-3 / FA-4 tiling keeps the L^2 term off HBM so the quadratic flops can run at tensor-core rate instead of at bandwidth rateFP8 attention doubles the arithmetic throughput and the ridge, so the compute-bound side starts at a longer context, not a shorter oneMoE deliberately lowers L* (Qwen3-30B-A3B at 13,824) - if your workload is attention-bound at 32k, an active-parameter-sparse model of the same size is more attention-bound, not less
notes
Design consequence: 'at what context length does attention dominate' has no single answer because it is a ratio of two widths in a config file, and it spans 13,824 to 104,448 across eleven architectures that all describe themselves as 'a transformer'. For a reader: read P_layer and n_q*head_dim out of the model's config and compute L* = P_layer/(n_q*head_dim) - there is nothing to measure and nothing to look up. Then multiply by the attention-layer fraction if the model is hybrid. WHERE THE DERIVATION BREAKS DOWN, six places. (1) It prices flops only. Attention at long context is often limited by KV BANDWIDTH rather than flops, and the flop crossover can occur well after the bandwidth crossover has already made the step slow - for Llama-3-8B on H100 with GQA-4 the ridge crossing is at L = 148 and the flop crossover at L = 53,248, a factor of 360 apart, so attention is compute-priced at 32k even though a 32k KV read is a lot of bytes. The two crossings bracket a real regime where prefill attention is compute-priced AND slow, which is the worst case and the one where kernel quality decides. (2) It counts a full causal mask, i.e. exactly L^2/2 useful score elements. Sliding windows, block-sparse patterns, and prefix-cache reuse all reduce the effective L^2 coefficient by more than the mask does - a prefix-cached prefill pays attention only on the uncached suffix. (3) It ignores that prefill flops are linear in L for the GEMM part but the ATTENTION part also includes the KV WRITE, which is linear in L and not quadratic; at small L the write is a larger share of attention's own cost than the L^2 term. (4) It assumes dense causal attention with no state passing. For SSM layers the per-token cost is a fixed scan, NOT L^2, so the hybrid correction above (dividing by the attention fraction) is only approximately right: the non-attention layers still contribute their own flops, so A(L) = L/(L+L*_eff) slightly OVERSTATES attention's share for hybrid models. I have stated the correction as an approximation rather than deriving it properly because doing so needs the Mamba-2 scan cost, which is a separate record. (5) head_dim is frequently not hidden_size/num_heads and must be read from the config, not inferred: Llama-3 needs d_h = 128 inferred (32 heads, d=4096 gives 128, consistent), Qwen3 sets head_dim explicitly to 128 with 32 heads on hidden 2048, giving n_q*d_h = 4096 = 2x hidden_size, and any model that inferred d_h = hidden/heads would get L* wrong by 2x. (6) GQA does not appear in L* at all, which is correct - attention flops depend on n_q and head_dim but not on n_kv - but this means the flop crossover is blind to the GQA ratio while the bandwidth crossover L_ridge depends on it directly. A reader who carries only L* will conclude GQA is irrelevant to attention cost, which is false; the two records have to be read together. One more caution on transferring these numbers: the L* values are properties of the released config, not of a checkpoint - fine-tunes that change intermediate_size or head count change L* proportionally, and no vendor publishes L* for a fine-tune. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.

Notes

Design consequence: 'at what context length does attention dominate' has no single answer because it is a ratio of two widths in a config file, and it spans 13,824 to 104,448 across eleven architectures that all describe themselves as 'a transformer'. For a reader: read P_layer and n_q*head_dim out of the model's config and compute L* = P_layer/(n_q*head_dim) - there is nothing to measure and nothing to look up. Then multiply by the attention-layer fraction if the model is hybrid. WHERE THE DERIVATION BREAKS DOWN, six places. (1) It prices flops only. Attention at long context is often limited by KV BANDWIDTH rather than flops, and the flop crossover can occur well after the bandwidth crossover has already made the step slow - for Llama-3-8B on H100 with GQA-4 the ridge crossing is at L = 148 and the flop crossover at L = 53,248, a factor of 360 apart, so attention is compute-priced at 32k even though a 32k KV read is a lot of bytes. The two crossings bracket a real regime where prefill attention is compute-priced AND slow, which is the worst case and the one where kernel quality decides. (2) It counts a full causal mask, i.e. exactly L^2/2 useful score elements. Sliding windows, block-sparse patterns, and prefix-cache reuse all reduce the effective L^2 coefficient by more than the mask does - a prefix-cached prefill pays attention only on the uncached suffix. (3) It ignores that prefill flops are linear in L for the GEMM part but the ATTENTION part also includes the KV WRITE, which is linear in L and not quadratic; at small L the write is a larger share of attention's own cost than the L^2 term. (4) It assumes dense causal attention with no state passing. For SSM layers the per-token cost is a fixed scan, NOT L^2, so the hybrid correction above (dividing by the attention fraction) is only approximately right: the non-attention layers still contribute their own flops, so A(L) = L/(L+L*_eff) slightly OVERSTATES attention's share for hybrid models. I have stated the correction as an approximation rather than deriving it properly because doing so needs the Mamba-2 scan cost, which is a separate record. (5) head_dim is frequently not hidden_size/num_heads and must be read from the config, not inferred: Llama-3 needs d_h = 128 inferred (32 heads, d=4096 gives 128, consistent), Qwen3 sets head_dim explicitly to 128 with 32 heads on hidden 2048, giving n_q*d_h = 4096 = 2x hidden_size, and any model that inferred d_h = hidden/heads would get L* wrong by 2x. (6) GQA does not appear in L* at all, which is correct - attention flops depend on n_q and head_dim but not on n_kv - but this means the flop crossover is blind to the GQA ratio while the bandwidth crossover L_ridge depends on it directly. A reader who carries only L* will conclude GQA is irrelevant to attention cost, which is false; the two records have to be read together. One more caution on transferring these numbers: the L* values are properties of the released config, not of a checkpoint - fine-tunes that change intermediate_size or head count change L* proportionally, and no vendor publishes L* for a fine-tune.

HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.

Cross-references

Hardware-dependent (4)