Let's Verify Step by Step

Even state-of-the-art models regularly produce logical mistakes, and it is unclear which of two supervision regimes - OUTCOME supervision (feedback on the final result only) or PROCESS supervision (feedback on each intermediate step) - is better and more cost-effective to collect.

Compares the two regimes directly. Process supervision labels each intermediate reasoning step as correct or not, enabling a PROCESS REWARD MODEL. Findings: process supervision significantly outperforms outcome supervision on the MATH dataset, and ACTIVE LEARNING (having the model itself flag its own uncertain steps for human labelling) markedly improves process-supervision efficiency, which matters because human feedback is the expensive input. Releases PRM800K, 800,000 step-level human feedback labels.

Process-supervised model solves 78% of problems from a representative subset of the MATH test set. Active learning significantly improves the efficacy of process supervision - i.e. it raises the LABEL-YIELD per unit of human effort, which is the cost number that matters, since human feedback is the bottleneck rather than GPU time.

ADOPTION: the PROCESS REWARD MODEL idea is in production - step-level verification is used in deployed reasoning systems (best-of-N over process-rewarded samples is one of the cheapest accuracy gains available), and PRM800K is a released dataset, so open_weights is recorded as true for the model/dataset artifacts rather than for a single checkpoint. VENUE NOTE: no peer-reviewed venue verifiable via Semantic Scholar; the arXiv comment carries no venue, so venue = arxiv-preprint, year = ArXiv v1 (May 2023). TIES THE TWO HALVES OF THIS SUB-AREA TOGETHER, which is why it is included in the inference-time-compute slice rather than the training slice: a PRM is what makes SAMPLED reasoning (self-consistency, best-of-N) pay off, and it is the EXTERNAL FEEDBACK that pap-eff-llms-cannot-self-correct-reasoning shows is missing in the intrinsic case. A step-level verifier costs an extra forward pass per step, so it is a real serving cost that belongs on the same accounting sheet as the reasoning tokens it screens. DANGLING REF STILL OPEN (2026-10-04 paper-read pass; supersedes the earlier 'LEFT UNRESOLVED' note, which reached the same conclusion but only from this record's own text - this time the PAPER itself was read). No A100, no GPU and no accelerator named anywhere in the paper; the experimental section describes models, datasets and sampling only (PRM800K, MATH, process vs outcome supervision). Source: https://arxiv.org/pdf/2305.20050 . Left dangling. VENUE CORRECTED 2026-10-04 (paper-verification pass): venue/year arxiv-preprint 2023 -> iclr 2024. EVIDENCE: Exact title "Let's Verify Step by Step" appears in the ICLR 2024 accepted-paper index at match score 1.0 (2275-title list, next best 0.435). Semantic Scholar returns "International Conference on Learning Representations". Same class as meth-detecting-pretraining-data: empty arXiv comment, no journal-ref, recorded as preprint despite peer-reviewed publication. This paper introduced the process-supervision PRM recipe that later inference work cites, so the venue matters for attribution. arxiv_id 2305.20050 is the preprint and is unchanged. Both values retained here deliberately; status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2305.20050). AFFILIATIONS BACKFILLED 2026-10-04 (paper-fields pass, agent w3p): PDF first page: all ten authors carry a single 'OpenAI' affiliation line, with a footnote marking the primary authors. OpenAlex returns none. Filled from the API/PDF rather than from memory, because an affiliation is a fact about a specific paper's author block and inference from the author's employer history is how wrong affiliations get shipped. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: iclr has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings. 2026-10-10: removed nvidia-a100-sxm. The paper names no accelerator at all; the experimental section describes models, datasets and sampling only (PRM800K, MATH, process vs outcome supervision). The ref was fabricated, not an ambiguous-capacity pick. 2026-10-10: removed nvidia-a100-sxm. The paper names no accelerator at all; the experimental section describes models, datasets and sampling only (PRM800K, MATH, process vs outcome supervision). The ref was fabricated, not an ambiguous-capacity pick.