MLPerf Inference Rules (official submission rules document)

CRITICAL FOR COMPARABILITY ANALYSIS. The official MLPerf Inference rules document (1230 lines) reveals structural reasons why vendor submissions may not be directly comparable: (1) CLOSED division requires same model as reference implementation, but OPEN division allows different models or retraining - so Open division results are NOT comparable across submitters. (2) Appendix C explicitly regulates speculative decoding: 'Implementations that artificially manipulate acceptance rates are disallowed, eg: using a different MTP head or continued pre-training of reference MTP head, quantization of the reference MTP head weights, post-training techniques on reference MTP head (like fine-tuning, RLHF etc.)' - this shows speculative decoding IS allowed in MLPerf but with constraints. (3) For LLM benchmarks, two latency metrics are collected: TTFT (time to first token) and TPOT (time per output token). (4) The rules allow 'additional inference parameters' in Open division that can significantly affect results. (5) Power measurement rules: 'The measured power is only valid for the accompanying benchmark. MLPerf Power is only capable of measuring and validating the full system power.' This means vendor power claims outside MLPerf are not validated.