inference/research
verifiedhigh · 0.95updated 2026-10-04

MLPerf Results Change Log - invalidated and modified results

database

Fields

id
sources/ev-mlperf-results-change-log
Publisher
MLCommons
Kind
database
Published
2024-01
Accessed
2026-10-04
notes
CRITICAL FOR MLPerf AUDITING. The official MLPerf change log documents that published results are sometimes modified or invalidated. Key findings: (1) Round 5.0 (Sept 2025): Google and NVIDIA results invalidated - 'Submitted as preview, but validation results were not submitted to available in the subsequent round.' (2) Round 4.0 (Aug 2024): Dell, HPE, Lenovo, Qualcomm results invalidated for same reason. (3) Round 4.1 (Apr 2025): AMD and UntetherAI results invalidated. (4) Round 4.0 Training: Multiple submitters (ASUSTeK, Dell, GigaComputing, NVIDIA, Quanta, Supermicro) had benchmark names changed from 'gnn' to 'RGAT' - indicating the original benchmark name was misleading. This log demonstrates that: (a) Preview results may never become available - they are not validated for production use. (b) Even published results can be invalidated post-hoc. (c) Benchmark names can be adjusted after publication, affecting comparability. This is direct evidence that MLPerf results require careful auditing before use in vendor comparisons.

Notes

CRITICAL FOR MLPerf AUDITING. The official MLPerf change log documents that published results are sometimes modified or invalidated. Key findings: (1) Round 5.0 (Sept 2025): Google and NVIDIA results invalidated - 'Submitted as preview, but validation results were not submitted to available in the subsequent round.' (2) Round 4.0 (Aug 2024): Dell, HPE, Lenovo, Qualcomm results invalidated for same reason. (3) Round 4.1 (Apr 2025): AMD and UntetherAI results invalidated. (4) Round 4.0 Training: Multiple submitters (ASUSTeK, Dell, GigaComputing, NVIDIA, Quanta, Supermicro) had benchmark names changed from 'gnn' to 'RGAT' - indicating the original benchmark name was misleading. This log demonstrates that: (a) Preview results may never become available - they are not validated for production use. (b) Even published results can be invalidated post-hoc. (c) Benchmark names can be adjusted after publication, affecting comparability. This is direct evidence that MLPerf results require careful auditing before use in vendor comparisons.

Cross-references

This record references no other records.