XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs

Agentic workloads need DYNAMIC structured generation - tool calls and response protocols that change both ACROSS requests and WITHIN a request - which breaks the assumption every existing structured-generation engine is built on: one fixed grammar for the whole output.

Extends the XGrammar design to structures that change mid-stream. Two named mechanisms: TAGDISPATCH for dynamic structural dispatching, where the current tag in the partial output selects which structure applies next, so a single decode can cross from a tool call into prose and back; and a CROSS-GRAMMAR CACHE for SUBSTRUCTURE-LEVEL reuse ACROSS grammars, so the shared primitives (a JSON key, a type declaration, a tool signature) are compiled once and reused by every structure that needs them - the reuse granularity moved from 'whole grammar' to 'substructure'. Three further efficiency mechanisms: an EARLEY-BASED ADAPTIVE TOKEN MASK CACHE, JUST-IN-TIME COMPILATION so only the parts of a grammar actually reached get compiled, and REPETITION STATE COMPRESSION for repeated structure.

Paper's own claim: over 6x faster compilation than prior structured-generation engines, and near-zero end-to-end overhead in modern LLM serving systems. Compilation speed is the right headline here precisely because just-in-time compilation is what makes dynamic structure affordable - the marginal cost of a NEW structure is compilation, so 6x on compilation is what determines whether an agent can use a per-request tool schema.

VENUE 'other' BECAUSE ACM CAIS IS NOT IN THE SCHEMA ENUM - THE AUTHORITATIVE VENUE IS ACM CAIS '26, read from the PUBLISHER'S OWN DEPOSIT rather than from arXiv. Crossref for DOI 10.1145/3786335.3813124 returns container-title 'Proceedings of the ACM Conference on AI and Agentic Systems', event 'CAIS '26: ACM Conference on AI and Agentic Systems' (San Jose CA USA), published-print 2026-05-26, pages 1009-1022, publisher ACM, type proceedings-article, alternative-id 10.1145/3786335 (the CAIS '26 volume). The arXiv comment reads only the acronym string 'ACM CAIS 26' and the Atom feed carries the DOI as journal_ref; recording the venue from the Crossref deposit rather than the acronym is the point, since a bare acronym in a comment field is a claim, not a venue. Crossref also supplies author AFFILIATIONS (Linzhang Li - Shanghai Jiao Tong University; Yixin Dong - Carnegie Mellon University), which the arXiv Atom feed does not carry at all. A future schema extension should add cais. WHY IT IS THE MOST DEPLOYMENT-RELEVANT PAPER IN THIS BATCH: the repo has no record for structured output beyond a benchmark, and by late 2026 the question a reader has is no longer 'can an engine enforce a JSON schema' but 'what does it cost when the structure CHANGES during one response' - which is the agentic-tool-calling case. XGrammar-1's static-grammar answer does not cover it; this does. Adoption in-upstream-engine on the same basis as XGrammar-1: the repo holds engines/sglang, and XGrammar-2 shares the sgl-project/xgrammar codebase. VENUE RECLASSIFIED 2026-10-05 (schema authority pass, agent w5-schema): venue 'other' -> 'cais', now that schemas/paper.schema.json carries a cais member. The enum member was added only because this record existed and could be reclassified in the same commit - adding a venue nobody records is how this repo shipped 36 dead enum values. The venue was verified against the PUBLISHER'S OWN Crossref deposit, not an arXiv comment string, because this repo has already been burned by comment-derived venues. AFFILIATIONS BACKFILLED FROM THE SAME DEPOSIT, because this record's own notes already flagged that Crossref carries them and the arXiv Atom feed does not: Linzhang Li (Shanghai Jiao Tong University), Yixin Dong (Carnegie Mellon University), Guanjie Wang (Shanghai Jiao Tong), Ziyi Xu (Shanghai Jiao Tong), Alexander Jiang (Carnegie Mellon), Tianqi Chen (Carnegie Mellon AND NVIDIA, Santa Clara). The record previously carried only ['Shanghai Jiao Tong University'], which dropped two of the three institutions - including NVIDIA, the one that matters for judging how much of the deployment story is vendor-neutral. The DEPOSIT detail that justifies 'cais' over 'other': container-title 'Proceedings of the ACM Conference on AI and Agentic Systems', event "CAIS '26: ACM Conference on AI and Agentic Systems", location San Jose CA USA, pages 1009-1022, issued and published-print 2026-05-26, parent volume 10.1145/3786335. Re-verified 2026-10-05 against the live Crossref record, which is unchanged.