The energy form: deriving joules per token from watts and throughput, and from bytes and bandwidth when you have neither, with a stated error bound

TWO ROUTES, AND THE CHOICE BETWEEN THEM IS THE WHOLE SUBJECT. ROUTE 1 (measured watts) has ZERO model error and should always be preferred: E/token [J] = P_mean [W] / throughput [tok/s]. The only requirements are that P_mean is the MEAN over the same window that produced the token count, that the window excludes the ramp, that the clock state is recorded, and that the power scope is named (board / system / node / rack). This is the whole content of the MLPerf power methodology, and the repo's H200 records are the worked example: 5,806.9 W mean node draw against 25,262.1 output tok/s gives 0.2299 J/output token exactly, and 5,854.3 W against 48,987.9 tok/s gives 0.1195 J/token for Mixtral-8x7B on the same node, same instrument, same locked 1000 MHz clocks. ROUTE 2 (derived, for a number you have not measured) is a three-step chain and every step has an error term you must state. Step 1, step time: t_step = bytes_step / BW_eff in the memory-bound regime, with bytes_step = w_b*P_eff + 2*d*V*w_b + kv_b*tok*L + kv_b*tok + overhead bytes (the first two terms and the crossover condition are derived in fct-batch1-step-cost-formula). Worked: Llama-3-8B bf16 on H100 SXM at batch 1, L = 1024 - weights 12.999 GiB (13.957e9 bytes), LM head 0.979 GiB, KV read 0.125 GiB, total 14.103 GiB = 1.514e10 bytes, t_step = 1.514e10 / 3.350e12 = 4.520 ms, so 221.2 tok/s, against a compute time of 15.2 us for the same step (memory-bound by 298x). Step 2, power: P = eta * TDP_board, and eta is the term everyone gets wrong because it is not 1 and not constant. The repo has two anchors and they disagree in an instructive way: an 8xH200 node plateau of 5,806.9 W against a GPU nameplate of 8 x 700 = 5,600 W gives eta_loaded = 1.037 - ABOVE nameplate, legitimately, because the node also carries two Xeon 8480C hosts, 2 TB DRAM, ten 400 GbE NICs and PSU losses that are not in the GPU nameplate; and an 8xH100 node under compute-intensive TRAINING at 76% of its 10.2 kW rating gives eta = 0.76 - BELOW nameplate, with a calibrated model reaching 11.4% MAPE against 27-37% for TDP-based estimation. The lesson is not a number, it is a direction: TDP understates a loaded multi-GPU NODE (non-accelerator overhead is outside the GPU nameplate) and overstates a single accelerator or a lightly loaded one. Step 3, utilisation, and this is the term most cost models omit entirely: with P_idle = 2,411.2 W, P_loaded = 5,806.9 W and full-rate throughput R = 25,262.1 tok/s from the SAME H200 node, linear interpolation gives P(u) = P_idle + u*(P_loaded - P_idle) and tok/s(u) = u*R, so E/token(u) = [P_idle + u*(P_loaded - P_idle)] / (u*R) = P_loaded/R + (P_idle/R)*(1/u - 1). The first term is the floor at full utilisation and the second is the entire penalty for running below it: u = 1.00 -> 0.2299 J (1.00x); u = 0.75 -> 0.2617 J (1.14x); u = 0.50 -> 0.3253 J (1.42x); u = 0.25 -> 0.5162 J (2.25x); u = 0.15 -> 0.7707 J (3.35x); u = 0.10 -> 1.0889 J (4.74x); u = 0.05 -> 2.0434 J (8.89x). It DIVERGES as u -> 0, which is the correct behaviour and not a modelling artefact: the repo's idle record has 2,411.2 W at literally 0 tok/s, so the naive E = P/throughput division is a division by zero and the naive E = u*P_loaded approach at u = 0.15 gives 871 W where the honest figure is 2,921 W - 3.4x wrong in the direction that makes an underbatched fleet look cheap. THE STATED ERROR BOUND, decomposed so a reader can see which term dominates: for a Route-2 estimate of joules per token, the error sources are (i) utilisation modelling, P_idle/P_loaded linear in u, worth up to 4.7x at u = 0.10 and unbounded below - DOMINANT, and the only fix is to measure at the intended utilisation; (ii) TDP-vs-actual, empirically 27-37% MAPE for TDP-based power estimation against 11.4% for a calibrated architecture-specific model, so a TDP-based joules figure carries roughly that relative error; (iii) peak-vs-sustained throughput, where peak is a best case at maximum batch and a real deployment runs 20-50% lower, and this error is one-sided (it makes joules per token look too LOW, because TDP overstates power while peak throughput overstates tokens and the two do not cancel); (iv) model-size transfer, since energy per token scales with ACTIVE parameters, not total, so scaling a measured figure by total parameters is wrong for every MoE and the repo's own pair demonstrates it - Mixtral-8x7B at 0.1195 J/token draws essentially the same watts as the dense 70B at 0.2299 J (5,854.3 vs 5,806.9 W, under 1% apart) because only ~12.9B of its 46.7B parameters are active, so the per-token efficiency ratio of 1.92x comes entirely from active-parameter sparsity; (v) power scope, where board-level to system-level is a 1.4-1.7x factor on datacenter nodes and a 20-50x factor on Apple silicon (4.4 mJ/token package against 0.09 J/token system for the same 3B model). ADDING (i)-(iv) IN QUADRATURE gives 30-50% for a TDP-based estimate at moderate utilisation, which matches the error bound already recorded in this repo, and the bound is only honest if the reader is told WHICH term dominates: at u < 0.25 the utilisation term alone exceeds the whole stated error bar. The cleanest single sanity check available to a reader with no meter: compare the model's implied bytes-per-joule against hardware bandwidth. 221.2 tok/s at, say, 700 W board gives 3.16 J/s per 221.2 tokens = 14.3 mJ/token, and 14.1 GiB per step at 3.35 TB/s means a memory-bound step CANNOT exceed about 238 tok/s on one H100 - so any joules-per-token figure implying more than 238 tokens/s from one H100 is arithmetically impossible and can be rejected without a power meter at all.

Design consequence: joules per token is not a hardware property, a model property, or a datasheet property - it is a property of a specific operating point, and the two largest error terms (utilisation and clock state) are both invisible in every published specification. The formula E/token(u) = P_loaded/R + (P_idle/R)*(1/u - 1) is the useful artifact here because it is derivable from two measured numbers on the same node and it immediately shows that the honest answer to 'how much does this token cost' depends on how full the machine is. A fleet at 15% utilisation pays 3.35x the per-token energy of the same fleet at full utilisation, which is the same conclusion as the repo's existing idle-draw gotcha but with the arithmetic attached. For hardware designers: the derivable quantity from bytes is FLOPS-per-second capped by BW, and energy per byte is the physically meaningful constant, but no vendor publishes it - the closest anchors are the H200 node at 5,806.9 W for 25,262 tok/s (0.2299 J/token, of which an unquantified share is the two Xeon hosts and the NICs) and Apple silicon at 4.4 mJ/token package-level against 0.09 J/token system-level, a 20x gap that is entirely measurement boundary. WHERE THE DERIVATION BREAKS DOWN, six places. (1) P(u) = P_idle + u*(P_loaded - P_idle) assumes power is linear in utilisation. It is not: DVFS means the clock, and therefore power-per-op, changes with load, and the repo records that a sustained load drives SW Power Cap and thermal throttling that move the tail of the latency distribution, that a 300 W cap on MI300X is silently enforced partly by dropping the MEMORY clock rather than the core clock, and that HW Power Brake cuts core clocks by 2x or more on a PSU assertion. Any of those puts the operating point on a different part of the power curve than the linear model assumes, in a direction that depends on the cap. (2) Throughput does not fall linearly with u either. At low batch the throughput curve is FLAT (batch 1 and batch 8 both give the same tokens/s, because the step is bandwidth-priced and the idle bandwidth is simply unused), so tok/s(u) = u*R is wrong in the region that matters most - the correct model has throughput flat from u=0 to u=B*-ish and rising after, which makes E/token(u) nearly CONSTANT below the crossover and then declining. The linear-interpolation formula above therefore OVERSTATES the utilisation penalty in the low-batch regime and UNDERSTATES it in the high-batch one; the 3.35x at u=0.15 is an upper bound, not a point estimate. (3) It prices only the accelerator path. The whole-node anchors include hosts, DRAM, NICs and PSU losses, so a board-only watts figure divided by whole-node throughput gives a systematically low joules figure for the GPUs and a correct one for the facility. (4) The Route-2 t_step ignores that real engines overlap weight streaming against compute, so the derived t_step is an upper bound on step time and the derived joules-per-token from Route 2 is an OVERESTIMATE - meaning Route 2 is conservative about cost, which is the safer direction but should be stated. (5) No figure here is portable across precision. The H200 MLPerf records explicitly do NOT state the served numeric format (weight_data_types fp16 with weight_transformations 'quantization, affine fusion' and no named served format), so 0.2299 J/token cannot be transferred to an fp8 run without a re-measurement, and the difference is likely to be substantial since fp8 halves the step's bytes at batch 1. (6) The Roofline t_step assumes 100% of peak bandwidth is achievable on the model's access pattern; the repo's own Marlin result (4x quantization speedup up to batch 16-32) is the evidence that real kernels land near but not on that line, and FA-2 reaching only 50-73% of peak FLOPS on A100 is the equivalent statement on the compute side. Finally, the arithmetic ceiling check is the one derivation that needs no power meter at all: on one H100 SXM a memory-bound Llama-3-8B decode step cannot exceed 1/4.520 ms = 221.2 tok/s, so a claimed 1,000 tok/s for a single-stream 8B model on one H100 is falsifiable from the datasheet alone. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.