id
gotchas/bench-batch-size-above-one-makes-the-batch-1-roofline-illegally-tight
Concepts
rooflinebatch-sizeaggregate-throughputper-user-throughputarithmetic-intensityfalse-positiveweight-amortisationcapacity-planning
Symptom
Run a whole-corpus roofline sweep that ignores batch size and it will condemn the repo's own best-documented measurement. The MPT-7B trio on 1x A100 40GB: MPT-7B in FP16 is 14 GB of weights, and the A100 40GB SXM4 streams 1555 GB/s, so the batch-1 decode roof is 1555e9/14e9 = 111.1 tok/s. mpt7b-a100-bs64-aggregate-output-tps records 800 tok/s - 7.2x the roof, which a sweep will report as the single most impossible number in the corpus. It is not impossible, it is the most ordinary number in it: at batch 64 the same 14 GB of weights are read ONCE per step and amortised over 64 in-flight sequences, so the weight stream stops being the binding constraint and the batch-1 roof simply does not apply. The per-user sibling, mpt7b-a100-bs64-per-user-decode-tps at 12.5 tok/s, is 11.3% of roof and is the figure a user actually experiences. Neither record is wrong. The sweep is wrong to compare either against a batch-1 ceiling without first checking the batch size, and 800 is the number most likely to be quoted.
Root cause
The batch-1 roofline is not a universal upper bound on decode throughput - it is the upper bound for ONE sequence, and it holds only while the weight stream dominates. Its exact statement is tok/s <= BW / weight_bytes at batch 1, and the reason it relaxes with batch is that the numerator is per-step and fixed while the denominator of useful work is the batch: at batch B a step delivers B tokens for the same 14 GB read, so the ceiling is B x 111.1 in the idealised memory-bound limit and in practice crosses into being compute-bound at some B*. This is the same crossover the repo's own fct-batch1-step-cost-formula record quantifies, and the same reason speculation dies as batch rises. The trap for a reader is that the roofline is famous as 'the hard physical limit' and is genuinely hard at batch 1, so applying it at batch 64 feels like rigour rather than error - and the failure is SILENT in the dangerous direction here, because an aggregate number that clears a physical ceiling invites the reader to conclude the hardware is better than physics allows, rather than concluding their own ceiling was the wrong one. The reciprocal hazard is the one the MPT-7B trio exists to document and which this record reinforces with arithmetic: aggregate rises 13.9x from batch 1 to 64 while per-user throughput FALLS 4.6x (57.6 -> 12.5). A record that reports the aggregate and lets a reader assume per-user inverts the conclusion about whether a deployment helps users.
Workaround
Never compute a roof without reading the batch size first, and treat 'batch size not a fixed published number' - the phrasing in most of the MLPerf records here - as ALSO not a fixed batch of 1. For a batch-1 record compute roof = BW/weight_bytes and a percentage of roof; for a record with batch or concurrency above 1, compute the roofline only as a per-GPU sanity check at the COMPUTE ceiling, which is batch-invariant: peak_decode_tok_s = PEAK_FLOPS/(2 x active_params). That second instrument is the one to reach for when sweeping aggregate records, because it does not need the batch size at all - it is what established that the whole MLPerf v6.1 corpus is legal, landing at 1.2%-18.5% of the FP4 compute roof. When a record must carry both conventions, carry both as SEPARATE records with the relationship derived in notes, as the MPT-7B trio does; that trio is the corpus's worked example and its arithmetic is now independently checked: aggregate 800/57.6 = 13.9x and per-user 12.5/57.6 = 4.6x worse, both matching what docs/07-benchmarking-methodology.md section 1 states.
notes
FOUND 2026-10-05 by the whole-corpus roofline sweep (agent w6-benchmarks), which hit this on its own reference case - the sweep produced 800/111.1 = 720% of roof for a record the repo holds up as its worked example of the per-user-vs-aggregate distinction, and the correct response was to fix the instrument rather than the record. THIS IS THE THIRD DISTINCT ROOFLINE TRAP IN THE CORPUS and the three are worth reading together because they fail in opposite directions. (1) This record: a batch-1 roof applied to an aggregate figure produces a FALSE POSITIVE on a compliant record. (2) bench-tok-s-derived-from-memory-bandwidth-is-not-a-benchmark: a fabricated tok/s figure produced by dividing bandwidth by a guessed bytes-per-token, which produces a FALSE NEGATIVE - the number looks plausible precisely because the divisor was made up. (3) The MoE active-versus-total divisor, which can produce either: applying a total-weight roof to Qwen3-30B-A3B or Qwen3.5-35B-A3B raises two known-good records to 314.5% and 336.6% of roof and looks like a finding when it is an artefact. THE INSTRUMENT VALIDATION THAT MAKES ALL THREE TRUSTWORTHY, run before any finding was reported: six batch-1 records whose byte counts the record itself states verbatim were checked against their own roofs and came in at 34.0% (llamacpp-radeon-r9700, 61.66 against 181.30), 43.0% (llamacpp-rx7900xtx, 47.06 against 109.43 - and that record's own notes independently measure 43.0% of combined DRAM peak, so the instrument reproduces a number the source measured by a different route), 82.5% and 9.0% (the two gpt-oss-120b MI50 records, 59.02 GiB stated verbatim), 51.9% (mpt7b-a100-bs1), and 60.9% (comm-5090-wsl2). Against that calibration the genuine violations stand: seven of the eleven deprecated local-* records exceed their roofs, reproducing and extending the five an earlier pass found, with local-rtx-pro-6000-70b-q8-decode at 230.1% the extreme. The compute-roof instrument, which needs no batch size, swept 33 aggregate records including every MLPerf v6.1 row and found ZERO violations - the highest is h200-8x-mlperf-v4-1-maxp at 55.4% and the lowest is 0.1% - which is the expected result for real submissions and is what makes the one Apple finding in bench-greenbench-apple-set-exceeds-its-own-part-roof credible rather than an artefact of the same shape.