Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Evaluating LLM chat assistants on open-ended questions is expensive and existing benchmarks don't measure human preferences well.

Uses strong LLMs (GPT-4) as judges to evaluate model outputs on MT-Bench (multi-turn questions) and Chatbot Arena (crowdsourced battles). Documents position bias, verbosity bias, and self-enhancement bias. Achieves >80% agreement with human preferences.

Published at NeurIPS 2023 Datasets and Benchmarks Track (verified from arXiv comments). This paper is foundational for LLM evaluation methodology but also illustrates the contamination problem: the same LLM-as-a-judge paradigm is now widely used, and the benchmarks themselves (MT-Bench, Chatbot Arena) are now potentially contaminated by training data. The paper documents biases (position, verbosity, self-enhancement) that must be controlled for. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.