inference/research
verifiedmedium · 0.75updated 2026-10-05

The overhead FLOP ledger: norms, RoPE, activations, softmax, residuals, KV-cache writes and sampling priced per decode step, fused versus unfused

memory bound

Fields

id
flops/fct-overhead-flop-ledger
Class
norm_softmax
Bound by
memory
Arithmetic intensity
THE HEADLINE IS A BYTE RESULT, NOT A FLOP RESULT, AND THAT IS THE POINT. The elementwise classes carry real flops but almost no arithmetic intensity, so they are invisible in a FLOP count and dominant in a bandwidth count. Per decode step, per token-slot, per layer, with d = hidden, I = intermediate, n_q = query heads, n_kv = KV heads, d_h = head_dim, and n_layers: RMSNorm x2 (input_layernorm and post_attention_layernorm): flops = 2 x 4d = 8d (d multiplies to form x^2, d adds, d multiplies by the gain weight, d multiplies by the normalized reciprocal-RMS), bytes = 2 x 3d x s (read x, read gain, write y), so AI = 8d/6d = 1.33 flop/byte at s = 2; residual add x2: flops = 2d, bytes = 2 x 3d x s (read two tensors, write one), AI = 0.33; RoPE on Q and K: elements = 2(n_q + n_kv)*d_h, flops ~ 1.5 per element for the rotate-half form (two multiplies and one add per rotated pair, and half the tensor is rotated), bytes = 2 x elements x s (read and write), AI = 0.19 - the WORST intensity of any class here, because RoPE is applied to a tensor two orders of magnitude smaller than the FFN and is almost always a separate kernel; SwiGLU activation: flops = 4dI for silu (exp, add, div, mul) plus dI for the gate multiply = 5dI against the FFN's own 6dI, i.e. the activation is 83% of the FFN's flops while being 0% of its bytes when fused into the GEMM epilogue; UNFUSED it costs 3 x dI x s bytes (read gate, read up, write the product) which at AI 0.83 is the single largest byte consumer in the entire decode step; softmax over scores: flops = 5 x L x n_q (max, subtract, exp, sum, divide), bytes UNFUSED = 2 x L x n_q x s (write the scores, read them back) = the L x n_q score matrix materialized twice, AI 0.625; with an online-softmax attention kernel those bytes are ZERO because the scores never leave SRAM/registers - this is the single biggest byte saving in the whole ledger and is what FlashAttention bought; KV cache write: flops = 0 exactly, bytes = 2 x n_kv x d_h x kv_bytes per layer per token - a pure store, 4,096 bytes per layer per token for GQA-8 at head_dim 128 and bf16, giving 128 KiB/token for Llama-3-8B, 320 KiB/token for Llama-2-70B, 256 KiB/token for Qwen3-32B, and 512 KiB/token for MHA Llama-2-7B; sampler (softmax + top-k + renormalize + draw): flops ~ 5V, bytes = 2 x 2 x V (fp32 logits read and write) at the fp32 cast vLLM performs unconditionally, AI ~1.25. LEDGER, ALL FOUR CLASSES, BATCH 1, L = 1024, bf16, with the step's weight stream for comparison (Llama-3-8B: 14.0 GF of GEMM flops against 15,009 MB of weight bytes; Llama-2-70B: 136.9 GF against 137,426 MB; Qwen3-32B: 60.4 GF against 61,954 MB): Llama-3-8B (d=4096, I=14336, 32/8 heads, d_h=128, 32 layers, V=128256) - RMSNorm x2 0.0010 GF / 1.573 MB / AI 0.67 (using the s=2 byte count: 2 x 3 x 4096 x 2 x 32 = 1.573 MB); residual x2 0.0003 GF / 1.573 MB / AI 0.17; RoPE 0.0005 GF / 2.621 MB / AI 0.19; SwiGLU 9.395 GF / 11,274 MB / AI 0.83; softmax 0.0052 GF / 8.389 MB / AI 0.63; KV write 0 GF / 0.131 MB; sampler 0.0006 GF / 0.513 MB / AI 1.25. UNFUSED TOTAL 9.40 GF against 11,289 MB = 75.2% of the step's HBM bytes, AI 0.83. FUSED TOTAL (SwiGLU into the GEMM epilogue, softmax inside the attention kernel) 9.40 GF against 6.28 MB = 0.04% of the step's bytes. Llama-2-70B (d=8192, I=28672, 64/8 heads, 80 layers, V=32000): UNFUSED 93.99 GF / 112,812 MB = 82.1% of the step's bytes; FUSED 27.66 MB = 0.02%. Qwen3-32B (d=5120, I=25600, 64/8 heads, 64 layers, V=151936): UNFUSED 41.97 GF / 50,383 MB = 81.3%; FUSED 17.91 MB = 0.03%. THE RESULT THAT MATTERS: left unfused, the elementwise classes are 75-82% of all the HBM bytes a batch-1 decode step moves - MORE than the weights. Fused, they are under 0.05%. This is not a marginal optimization; it is the difference between a decode step that reads 14 GiB and one that reads 26 GiB. The FLOP side is the mirror image and must not be conflated: the overhead classes carry 67-69% as many flops as the step's GEMMs (9.40 GF against 14.0 GF for Llama-3-8B, 94.0 against 136.9 for the 70B, 42.0 against 60.4 for Qwen3-32B), almost all of it in SiLU's exp and the softmax's exp. Those flops execute on the ALU and SFU, not on the tensor cores, at roughly 1/30th the bf16 tensor-core rate on H100, so 69% more flops that run 30x slower contribute about 2% of the step's time when memory-bound - and become the dominant term when the step is compute-bound, which is the regime where a fused, SFU-friendly activation kernel is worth having. Batch invariance: every one of these terms is linear in B, so the overhead's SHARE of the step is invariant to batch (Llama-3-8B at B = 64: UNFUSED 601.8 GF / 722,461 MB = 75.2%, identical to B = 1). That invariance is the useful structural fact: fusion decisions taken at batch 1 remain correct at batch 256, but the overhead's share of BYTES only falls if the FLOP side of the ledger grows faster than its byte side, which happens only when you move from elementwise kernels into a compute-bound regime.
Scales with
fused_vs_unfused (the dominant lever by a factor of 1800 on bytes: 11,289 MB unfused vs 6.28 MB fused on Llama-3-8B)intermediate_size (SwiGLU's byte cost is 3*d*I*s per layer, the largest single overhead byte term when unfused)hidden_width (every norm, residual and rope term is linear in d)num_attention_heads x head_dim (RoPE and softmax terms)vocab_size (the sampler, and the fp32 cast that doubles it)context_length (softmax's L x n_q materialization; ZERO under FlashAttention)kv_dtype_bytes (the KV write, which is the only overhead term with zero flops)batch_size (all terms linear - the overhead share is batch-invariant)
Hardware parameters
parametertensor_core_counteffectThose flops execute on the ALU and SFU, not on the tensor cores, at roughly 1/30th the bf16 tensor-core rate on H100, so 69% more flops that run 30x slower contribute about 2% of the step's time when memory-bound - and become the dominant term when the step is compute-bound, which is the regime where a fused, SFU-friendly activation kernel is worth having.basisrecordedmagnitude30xvalues_fornvidia-h100-sxm,amd-instinct-mi300x,nvidia-b200,nvidia-rtx-4090,nvidia-l4
Workarounds
Fuse SwiGLU into the preceding GEMM's epilogue (gate and up are one GEMM, the activation is the epilogue) - this removes 75-82% of the step's overhead bytes in one changeFuse softmax into the attention kernel as an online-softmax accumulator so the L x n_q score matrix never reaches HBM (arXiv 2205.14135) - removes the second-largest overhead byte termFuse RMSNorm+quantization or RMSNorm+matmul so the norm's output never round-trips HBMFuse RoPE into the QKV projection's epilogue or into the attention kernel's Q/K loadPersistent read-once-write-once norm kernels instead of naive elementwise launchesFused samplers that read the logits once and do top-k/top-p/renormalize/draw in registers (see sampling, host-side-sampling-loop)Keep the logits in 2-byte form through the sampler unless fp32 is required - vLLM's unconditional logits.to(torch.float32) doubles the V-wide traffic on both device and hostSkip the KV-cache write entirely for rejected speculative tokens, and write fp8/fp4 KV where quality allows, halving the one zero-flop overhead term
notes
Design consequence: the ledger shows there are two distinct overhead problems that are routinely conflated. The BYTES problem is real now, at batch 1, on every model: elementwise work is 75-82% of a decode step's HBM traffic if left unfused, and fusing it is worth a large multiple of any kernel-level improvement. The FLOP problem is real later, above B*, where 69% more flops at 1/30th the tensor-core rate becomes a first-order term. Saying 'overhead flops are a few percent' is wrong on the flops and right on the time, and saying 'overhead is a few percent of bytes' is wrong by two orders of magnitude. For a reader: the ledger above is the complete per-step inventory of what does not touch a tensor core, and its own bytes column tells you which fusion to write first - SwiGLU first (3*d*I*s), softmax second (2*L*n_q*s), everything else is rounding error by comparison. WHERE THE LEDGER BREAKS DOWN, six places. (1) The SwiGLU flops of 4 per element for silu is an approximation: silu is sigmoid-then-multiply, and the sigmoid itself is roughly 4 SFU ops, but real implementations use a fast approximation with a different op count, and the exp/div used may be hardware reciprocal-unit instructions rather than full divides. A different activation (GELU, ReLU2 - Nemotron-H uses relu2, which is 1 flop) changes these numbers by an order of magnitude, so this ledger is a SWIGLU ledger, not an activation ledger. (2) The RoPE count of 1.5 flops per element assumes the rotate-half formulation; the interleaved/NeoX formulation differs, and MLA applies RoPE to only the 64-wide rope sub-head of a 192-wide head, so a DeepSeek-V3 RoPE cost is roughly a third of the naive estimate. (3) It assumes the KV cache write is one contiguous store per layer per token. PagedAttention writes into non-contiguous blocks via a scatter, which is measured elsewhere in this repo to be silently corruptible under CUDA graphs when written in Python - so the BYTES are right but the COST of achieving them is not captured. (4) It ignores attention's own read traffic entirely, which belongs to decode_attention not here, and at L = 1024 on GQA-8 that traffic (4*L*n_kv*d_h*s per layer = 4.19 MB per layer, 134 MB over 32 layers) is larger than everything in this ledger except SwiGLU - so at long context this ledger is NOT the whole overhead story even fused. (5) The AI values assume perfect coalesced access and that each tensor is touched once. Fused kernels keep intermediates in registers and never touch HBM at all, which is why 'FUSED' rows show zero bytes for some terms; a partially fused kernel that writes the activation out to score a later op pays the bytes back. (6) The FLOP counts are counts, not time. Mapping them to time requires knowing the SM's SFU throughput and its ALU-to-tensor-core ratio, which is not in any accelerator record in this repo and is not derivable from the published TFLOPS figure. I have therefore stated the 1/30th figure as an order-of-magnitude remark about H100 and NOT used it in any derived number - the flops are recorded, the time conversion is left to the reader with a warning. Finally, the launch-overhead dimension is entirely outside this ledger: these classes are also the classes whose unfused form costs a separate kernel LAUNCH each, and at batch 1 the sum of those launches is a latency floor that no amount of arithmetic accuracy in this ledger predicts - see custom-kernel-launch-overhead and sched-iteration-level-scheduling. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.

Notes

Design consequence: the ledger shows there are two distinct overhead problems that are routinely conflated. The BYTES problem is real now, at batch 1, on every model: elementwise work is 75-82% of a decode step's HBM traffic if left unfused, and fusing it is worth a large multiple of any kernel-level improvement. The FLOP problem is real later, above B*, where 69% more flops at 1/30th the tensor-core rate becomes a first-order term. Saying 'overhead flops are a few percent' is wrong on the flops and right on the time, and saying 'overhead is a few percent of bytes' is wrong by two orders of magnitude. For a reader: the ledger above is the complete per-step inventory of what does not touch a tensor core, and its own bytes column tells you which fusion to write first - SwiGLU first (3*d*I*s), softmax second (2*L*n_q*s), everything else is rounding error by comparison. WHERE THE LEDGER BREAKS DOWN, six places. (1) The SwiGLU flops of 4 per element for silu is an approximation: silu is sigmoid-then-multiply, and the sigmoid itself is roughly 4 SFU ops, but real implementations use a fast approximation with a different op count, and the exp/div used may be hardware reciprocal-unit instructions rather than full divides. A different activation (GELU, ReLU2 - Nemotron-H uses relu2, which is 1 flop) changes these numbers by an order of magnitude, so this ledger is a SWIGLU ledger, not an activation ledger. (2) The RoPE count of 1.5 flops per element assumes the rotate-half formulation; the interleaved/NeoX formulation differs, and MLA applies RoPE to only the 64-wide rope sub-head of a 192-wide head, so a DeepSeek-V3 RoPE cost is roughly a third of the naive estimate. (3) It assumes the KV cache write is one contiguous store per layer per token. PagedAttention writes into non-contiguous blocks via a scatter, which is measured elsewhere in this repo to be silently corruptible under CUDA graphs when written in Python - so the BYTES are right but the COST of achieving them is not captured. (4) It ignores attention's own read traffic entirely, which belongs to decode_attention not here, and at L = 1024 on GQA-8 that traffic (4*L*n_kv*d_h*s per layer = 4.19 MB per layer, 134 MB over 32 layers) is larger than everything in this ledger except SwiGLU - so at long context this ledger is NOT the whole overhead story even fused. (5) The AI values assume perfect coalesced access and that each tensor is touched once. Fused kernels keep intermediates in registers and never touch HBM at all, which is why 'FUSED' rows show zero bytes for some terms; a partially fused kernel that writes the activation out to score a later op pays the bytes back. (6) The FLOP counts are counts, not time. Mapping them to time requires knowing the SM's SFU throughput and its ALU-to-tensor-core ratio, which is not in any accelerator record in this repo and is not derivable from the published TFLOPS figure. I have therefore stated the 1/30th figure as an order-of-magnitude remark about H100 and NOT used it in any derived number - the flops are recorded, the time conversion is left to the reader with a warning. Finally, the launch-overhead dimension is entirely outside this ledger: these classes are also the classes whose unfused form costs a separate kernel LAUNCH each, and at batch 1 the sum of those launches is a latency floor that no amount of arithmetic accuracy in this ledger predicts - see custom-kernel-launch-overhead and sched-iteration-level-scheduling.

HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.

Cross-references

Hardware-dependent (5)