Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (arXiv:2407.21787)

arXiv v3 (Dec 2024). Explores inference compute as a scaling axis via repeated sampling. Coverage scales log-linearly with number of samples over four orders of magnitude. On SWE-bench Lite, DeepSeek-Coder-V2-Instruct increases from 15.9% (1 sample) to 56% (250 samples), outperforming single-sample SOTA of 43%. Finding: majority voting and reward models plateau beyond several hundred samples and fail to fully scale with sample budget.