From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

Manual curation of high-quality LLM benchmarks is expensive and doesn't scale.

BenchBuilder: automated pipeline using LLMs to curate prompts from crowdsourced data (Chatbot Arena, WildChat-1M). Arena-Hard-Auto: 500 challenging prompts with 3x higher separation than MT-Bench, 98.6% correlation with human preference rankings, at $20 cost.

arXiv preprint, no venue claim in comments. This paper directly addresses benchmarking criticism: it shows that MT-Bench has low separation between models and proposes a more discriminative alternative. The 3x higher separation claim is the paper's own measurement. The $20 cost is for LLM-as-a-judge evaluation. VENUE (+YEAR) CORRECTION AND VENUE_TRACK 2026-10-04 (paper-fields pass, agent w3p): venue 'arxiv-preprint' -> 'icml', venue_track 'main'. EVIDENCE: PMLR v267 (ICML 2025) table of contents contains the title, with the proceedings casing 'From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline' - the record's name capitalises 'High-Quality' and the arXiv title differs slightly. v267 confirmed as the 42nd ICML, 13-19 July 2025. NOTE THE YEAR MOVES 2024 -> 2025: the arXiv posting is 2024 but the venue year is 2025, which is why Semantic Scholar returned 2024 for it. BOTH VALUES RETAINED per the repo contract: the prior value was 'arxiv-preprint'. status set to contested because this changes a peer-reviewed attribution. Method note: OpenReview's api2 hard-429s under load and pre-2024 venues are unreachable through it, so these were verified against static proceedings indexes (proceedings.mlr.press for ICML, proceedings.mlsys.org for MLSys) rather than the OpenReview API. Year also changed 2024 -> 2025; the prior value 2024 is the arXiv posting year and remains true of the preprint. 2026-10-10: PMLR v267 ICML 2025 proceedings (idx2.py: https://proceedings.mlr.press/v267/) confirms "From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline" (Tianle Li et al.). Source meth-arena-hard-benchbuilder-arxiv cites arXiv preprint vs PMLR v267 ICML 2025. Sources disagree; status remains contested.