SGLang: Efficient Execution of Structured Language Model Programs

Complex LLM applications chain many generation calls with control flow, shared state and structured I/O, but serving runtimes execute each call independently and waste the reuse across them.

A frontend language with primitives for generation, parallel calls, branching and state, feeding a runtime with RadixAttention: a prefix tree over the KV cache that reuses cached prefixes across requests sharing a system prompt or few-shot prefix, instead of recomputing them.

Up to 6.4x higher throughput than state-of-the-art inference systems, per the arXiv abstract, on large language and multimodal models across agent control, logical reasoning, few-shot learning benchmarks, JSON decoding, RAG pipelines and multi-turn chat. The 6.4x is a best case over a suite of structured programs; single-turn chat has far less shared prefix to reuse and the gain is much smaller. No baseline engine is named in the abstract itself — read the paper's evaluation section for which systems were compared.

SLUG RENAMED 2026-10-04 (parent agent): was 'sglang', colliding with the engines record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs (['affects', 'engine_id']) all mean the engines record, so none needed repointing; papers are cited via `sources`, never by bare slug. Venue verified two independent ways: Crossref (NeurIPS 2024 proceedings, DOI 10.52202/079017-2000) and OpenReview (venue 'NeurIPS 2024 poster'). Adoption is genuinely in-production: sgl-project/sglang is the repo behind the deployed SGLang serving framework, and RadixAttention is its own shipped mechanism rather than a patch on someone else's engine. This is the strongest 'citation means shipped' case in the slice after vLLM. DUPLICATE COVERAGE WARNING: another agent's record with a different slug covers the same paper ('sglang-structured-program-runtime', arXiv 2312.07104, sources flop-sglang). This record is the verified one: its venue is confirmed against a proceedings/DOI/OpenReview source, whereas that record lists the venue as arxiv-preprint, i.e. unverified. The two should be merged or one deleted by whoever owns the merge; I did not delete the other record because it is outside my write scope. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings. DUPLICATE-SLUG-CONSOLIDATION-2026-10-05 (agent w6-papers). data/papers/paper-sglang.json and data/papers/sglang-structured-program-runtime.json were BOTH live records for arXiv 2312.07104 (SGLang: Efficient Execution of Structured Language Model Programs), so one paper appeared twice under two slugs and - where the records disagreed - under two different venues. sglang-structured-program-runtime.json has been merged into this record and removed. IDENTIFICATION IS BY ARXIV ID, RE-FETCHED NOT ASSUMED: the arXiv API returns one entry for 2312.07104 with this title and the author list [Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Yuandong Wang, You Zhao, Zihao Jiang, Zilin Miao, Shang Yan, Shiqi He, Kaijie Fan, Yang Bai, Hai-Chao Wang], which both records were matching. Source w6p-arxiv-api-duplicate-id-confirmation. Venue evidence per record is in w6p-openreview-api2-duplicate-venue-records. Every arXiv id in this consolidation was taken from an API response rather than from memory or from the records, per the rule that this repo has shipped wrong ids twice. WHAT WAS CHECKED BEFORE REMOVING ANYTHING: every field of every data/**/*.json record was scanned for the string "sglang-structured-program-runtime". Zero inbound references were found - no model, gotcha, source, engine or doc record pointed at it - so the merge breaks no reference. Papers are cited by SOURCE ID, never by slug, so no `sources` entry needed repointing; the retired record's source ids were folded into this record's `sources` so the evidence survives the merge. MERGED FIELD BY FIELD - nothing was silently dropped: mechanism: the retired record's text was materially fuller (678 chars vs 276) and is RETAINED here rather than discarded. Its text: A frontend language plus a runtime. Frontend provides primitives for generation and explicit parallelism control, so programs are not implicitly sequential. Runtime contributes two optimisations that matter for this slice: RADIXATTENTION, which organises the KV cache in a radix tree over token sequences so a shared prefix across calls is matched and reused rather than recomputed (this is the prefix-reuse mechanism behind agentic multi-turn serving), and COMPRESSED FINITE STATE MACHINES for faster structured-output decoding, which collapses grammar/constraint automaton states into a compact representation so the per-token mask computation on the critical path is cheaper. speedup_reported: the two records carried DIFFERENT text for the same paper. This record's text is retained and the retired record's variant is quoted so the difference is visible: "Up to 6.4x higher throughput than state-of-the-art inference systems, across various large language and multimodal models, on tasks INCLUDING agent control, logical reasoning, few-shot learning, JSON decoding, retrieval-augmented generation pipelines, and multi-turn chat. The headline is a THROUGHPUT number against a baseline serving system, on a program-level workload - not a batch-1 latency multiplier, which makes it directly relevant to this repo's benchmark records rather than only comparable to other systems papers." AFFILIATIONS CORRECTED - union taken rather than resolved from the paper. This record listed Stanford and the LMSYS Organization; the retired twin listed Stanford, CMU, UC Berkeley, Shanghai Jiao Tong and Texas A&M. The twin's list is KEPT because it is the richer one and dropping four institutions on a hunch would lose real information - but it is asserted by the twin and was NOT independently re-verified from a publisher deposit in this pass, so it is flagged as such. No affiliation here is contradicted by anything read during this wave. The arXiv API author list for 2312.07104 confirms the twelve named authors in the retired record are exactly the v2 byline, which corroborates that record's provenance even where its affiliations are unverified. VENUE AND THE CLAIMS THAT COMPETED FOR IT: NeurIPS 2024 on both records; venue was never the problem here, the DUPLICATION was. Verified two independent ways: the Crossref NeurIPS 2024 proceedings deposit, and the OpenReview note with venue 'NeurIPS 2024 poster' / venueid 'NeurIPS.cc/2024/Conference'. One field could not be resolved and is flagged rather than averaged: the two records disagreed on adoption, this record saying in-production and the retired twin saying in-upstream-engine. Both are defensible under the schema's own definitions - SGLang is a real production-grade serving engine that people deploy, which is in-production, while the schema's adoption field is meant to record the METHOD's adoption and SGLang is an engine rather than a feature one enables, which is in-upstream-engine. This record's value is retained; the twin's reading is preserved here so the choice is visible rather than silent. STATUS: the contradiction this pair represented is now resolved, so the record is no longer marked contested on account of the duplicate. Fields were changed as stated above; arxiv_id is unchanged. SUPERSEDED EARLIER FLAG: this record previously carried a note beginning "DUPLICATE COVERAGE WARNING..." which flagged the slug-pair venue disagreement and explicitly deferred fixing it to a later pass. That deferral is discharged by this merge - the disagreement no longer exists because there is only one record.