Medusa speedup degradation beyond batch size 32

Simulated on A100 80GB PCIe, sequence length 1024. Speedup measured across batch sizes 1-64.

Medusa paper Appendix G.3: 'when the batch size exceeds 32, the speedup decreases and may even have a negative effect.' This occurs because linear layers shift from memory-bandwidth-bound to compute-bound. At batch 1, Medusa achieves >2.2x speedup; at batch 32+, the speedup degrades and can become negative (slower than baseline). This is the key measured evidence for the batch dilution problem. LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: NOT STATED by the source - absent, not inferred. A throughput figure with no scenario cannot be placed against an MLPerf Offline or Server number (they are different measurements; on identical hardware the Offline-minus-Server difference is negative in 5 of 9 systems measured). ABSENT CONDITIONS (4 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - engine version: unrecoverable from sd-medusa-paper - engine_id=vllm is named but no build/commit/container digest is published - input length: unrecoverable from sd-medusa-paper - would need the run's ISL or the prepared dataset's input-length profile - output length: unrecoverable from sd-medusa-paper - would need the run's OSL or the harness's configured max_tokens - warmup: unrecoverable from sd-medusa-paper - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted. UNIT/METRIC DIRECTION CORRECTION 2026-10-04 (benchmark throughput + latency audit). The stored value is UNCHANGED; what was wrong is the `metric` enum, which labelled this record as a raw tokens/s measurement when the number is a dimensionless RATIO. Stored value is 0 with unit `speedup_ratio`: the paper measures that beyond batch size 32 the speedup DECAYS toward zero and can go negative, and 0 is the floor of that range, not a throughput measurement of zero tokens/s. Reading `metric: decode_tok_s, value: 0` as a measured decode rate is wrong in a way that matters - it looks like a failed run, not a documented degradation curve. This is the error docs/07-benchmarking-methodology.md section 8 warns about from the other side: a reader filtering data/benchmarks/ by `metric: decode_tok_s` expects tokens per second, so a speedup_ratio filed under that key is not a wrong number, it is a number of the wrong KIND in the one field a capacity planner is most likely to filter on. CORRECT ENUM MEMBER: none of decode_tok_s / prefill_tok_s / tps_aggregate / tok_s_per_user describes a ratio, and schemas/benchmark.schema.json is out of scope for this audit so the record is left with an explicit `unit` and this note rather than silently reclassified. Anyone selecting benchmarks by metric MUST additionally check `unit` until a speedup_ratio member exists in the enum. METRIC CORRECTION 2026-10-05 (schema authority pass, agent w5-schema). The stored VALUE is UNCHANGED. What changed is `metric`, which labelled a DIMENSIONLESS RATIO as a dimensioned rate. schemas/benchmark.schema.json now has a speedup_ratio member, so the number no longer has to borrow a rate key it does not belong under, and every filter on data/benchmarks/ that selects tok/s no longer returns a ratio. Value and baseline unchanged; only the kind of quantity is now stated by the field that means it. BREAK-EVEN CHECK 2026-10-05 (agent w6-benchmarks): VALUE UNCHANGED, and the existing explanation of the 0 is CORRECT and survives the arithmetic independently. A zero speedup past batch 32 is the expected sign, not an anomaly: speculation is priced at A_be = 1/k while memory-bound and at approximately 1 once compute-bound, so a scheme that cleared break-even trivially at batch 1 (Medusa's paper states >2.2x there) is expected to decay toward zero and then below it as the batch crosses the roofline crossover. The 0 here is a real measurement of a curve reaching its floor, not a missing value. This is the only record in data/benchmarks/ where a stored 0 is the correct reading of a number rather than a hole, and the metric=speedup_ratio filing is what makes that legible. ACCELERATOR_IDS BACKFILL 2026-10-08 (w10-bench-formatid): the accelerator_ids field was absent and is now ['nvidia-a100-80gb-pcie']. EVIDENCE, quoted from this record's own text: methodology: "Simulated on A100 80GB PCIe, sequence length 1024"; the unit's BASELINE sentence names the same part; nvidia-a100-80gb-pcie exists in data/accelerators/. NO NUMBER CHANGED. The other 22 records with no accelerator_ids name no accelerator at all, or name one with no record in data/accelerators/ - that is a gap to report, not a value to invent, so they stay empty. ACCELERATOR_IDS BACKFILL 2026-10-08 (w10-bench-formatid): the accelerator_ids field was absent and is now ['nvidia-a100-80gb-pcie']. EVIDENCE, quoted from this record's own text: methodology: "Simulated on A100 80GB PCIe, sequence length 1024"; the unit's BASELINE sentence names the same part; nvidia-a100-80gb-pcie exists in data/accelerators/. NO NUMBER CHANGED. The other 22 records with no accelerator_ids name no accelerator at all, or name one with no record in data/accelerators/ - that is a gap to report, not a value to invent, so they stay empty.