Widely-cited benchmarks can measure the wrong thing: MT-Bench has low separation, LLM-as-a-judge has biases, and contaminated benchmarks measure memorization

You cite a benchmark result (e.g., 'Model X scores 85 on MMLU') as evidence of capability, but the benchmark is contaminated, has low separation between models, or measures something other than what you care about.

Three independent problems: (1) Contamination: models have seen test data in training (Shi et al. 2023). (2) Low separation: MT-Bench can't distinguish between models that are actually different (Li et al. 2024, Arena-Hard). (3) Judge bias: LLM-as-a-judge has position, verbosity, and self-enhancement biases (Zheng et al. 2023). All three make benchmark results less meaningful.

Before citing a benchmark: (1) Check if the benchmark is likely contaminated (is it popular? is it on the internet?). (2) Check if the benchmark has good separation (can it distinguish between models?). (3) If using LLM-as-a-judge, control for biases (randomize order, use multiple judges). (4) Prefer benchmarks that are continuously updated and have anti-contamination measures.

The Arena-Hard paper (Li et al. 2024) shows MT-Bench has 3x lower separation than Arena-Hard-Auto. The LLM-as-a-judge paper (Zheng et al. 2023) documents systematic biases. The contamination paper (Shi et al. 2023) shows detection is possible. Together, these three papers form the core of the benchmark validity crisis.