JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models

Constrained decoding has standardised on JSON Schema and grown in adoption, yet nobody has systematically evaluated whether the shipped frameworks actually WORK on real-world schemas, or what they cost.

Not an inference method - an evaluation framework. Assesses constrained decoding along three dimensions: efficiency in generating constraint-compliant output, COVERAGE of constraint types (can the framework express them at all?), and QUALITY of what it emits. Ships JSONSchemaBench: 10K real-world JSON schemas spanning a wide range of constraints at varying complexity, paired with the official JSON Schema Test Suite for conformance checking. Evaluates six state-of-the-art frameworks: Guidance, Outlines, Llama.cpp, XGrammar, OpenAI, and Gemini - i.e. the actual production constrained-decoding stacks, not research prototypes.

Not a speedup paper. Its finding is the one that matters for this repo: coverage of constraint types is INCOMPLETE across all six frameworks, so 'supports JSON Schema' is not a sufficient claim - a schema feature unsupported by the chosen engine fails silently or at request time. No single latency multiplier is claimed in the abstract and none is recorded here.

ADOPTION of the BENCHMARK: released publicly and usable against any constrained-decoding stack, which is why this record is the evaluation-infrastructure entry for the structured-output sub-area. THE KEY CROSS-RECORD FACT: this benchmark is the only place in the repo that evaluates Outlines, Guidance, XGrammar, and llama.cpp's grammar path SIDE BY SIDE on the same schemas - and Outlines and Guidance do NOT have arXiv preprints at all (both were searched for exhaustively via the arXiv API, by title, by author, and via the arXiv search UI; neither resolves), which is exactly why a benchmark paper matters more for them than a paper would. Every claim about structured-output engine coverage should be routed through this record rather than to a vendor page. VENUE NOTE: no venue verifiable via Semantic Scholar and no arXiv comment venue, so venue = arxiv-preprint, year = ArXiv v1 (Jan 2025). hardware_relevance is empty because the benchmark measures framework overhead and correctness, not hardware-dependent speedup. AFFILIATIONS BACKFILLED 2026-10-04 (paper-fields pass, agent w3p): OpenAlex institutions field (EPFL); the PDF was not parsed for this one. Filled from the API/PDF rather than from memory, because an affiliation is a fact about a specific paper's author block and inference from the author's employer history is how wrong affiliations get shipped. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.