Guidance
transformers, llama.cpp, openai, cuda, cpu
Fields
id
engines/niche-guidanceRepo
guidance-ai/guidance
Languages
PythonNotebookHTMLTypeScript
License
MIT
First release
2022
Design goal
structured-outputease-of-useresearchprompt-and-constraints-together
Scheduling
none
Backends
transformersllama.cppopenaicudacpu
Formats
safetensorsgguf
Notable features
A different mechanism from every other structured-generation library here. Guidance constrains generation AND allows backtracking over already-generated tokens by re-masking, rather than only restricting the next-token logits. That is why it can do prompt-plus-constraint interleaving in one passConstraint system covers any context-free grammar, so long as the backend model has full Guidance support (the README's own qualifier)Programming-model API: `lm += gen(regex=...)` writes the constraint as part of the program, so constraints are interleaved with generation and with each otherBackend-agnostic by design - README names Transformers, llama.cpp and OpenAI, and the guiding principle is 'if you already have the backend required for your model, you can simply run pip install guidance'Mock backend for testing: the README highlights local validation of candidate strings and testing full runs with the Mock model, which is the cheapest way to develop a constrained program without a GPUMIT-licensed, 21787 stars - the largest community of the three structured-generation librariesLast push 2026-05-21, release 0.3.2 on 2026-03-18, 633 open issues
Hardware caveats
The backtracking mechanism has a real cost the README does not quantify: re-masking and re-running already-generated tokens means recomputation. On a long constrained generation this can be a large multiple of a single forward pass, and the README gives no overhead table - unlike XGrammar, which at least claims near-zeroIt is a library, not an engine. scheduling is 'none' and the backend list reflects which host backends Guidance can drive, not backends it implementsFull grammar support requires the backend to have 'full support for Guidance'. Against a plain OpenAI-compatible HTTP endpoint you get remote generation without the re-masking machinery, so the interesting features degradeIt is NOT the default structured backend in any major serving engine. vLLM's default is XGrammar (see [[niche-xgrammar]]). If you want Guidance's semantics in production you are either in-process or writing your own integrationIts own README comparison, as rendered in [[niche-lm-format-enforcer]]'s table, marks Guidance as lacking batched generation and beam search and not integrating into existing pipelines - both are real limits on using it in a serverRelease cadence has slowed: 0.3.2 shipped 2026-03-18 and the last push was 2026-05-21, roughly five months before this record. Still maintained, not urgentThe repo's language breakdown is dominated by Jupyter Notebooks (29MB) over Python (884KB) - it is a research-and-teaching project that happens to be usable in production, and the docs reflect that
Best for
Writing programs that interleave constraints with generation, which no other library in this slice does - regex fields appearing mid-output, conditions on later fields constraining earlier onesIterating on a constrained prompt cheaply using the Mock backend, with no GPUStructured extraction with a grammar you control fully rather than a JSON schema subsetReading the clearest statement of why re-masking-based steering and logit-restriction-based constraint enforcement are different problems
Avoid for
Production per-token paths in a serving engine - use the engine's default structured backendBatched generation or beam search, which its peers' comparison tables mark as unsupportedRemote HTTP backends where you need the re-masking featuresLatency-critical generation, given the unquantified recomputation cost of backtracking
notes
Fourth of the structured-generation records (with [[niche-xgrammar]], [[niche-outlines]], [[niche-lm-format-enforcer]]) because it is the one with a genuinely different mechanism: it constrains and backtracks rather than only restricting logits. That mechanism is why it survives alongside the logit-restriction libraries rather than being displaced by them. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-05-21 (4.5 months before this record), latest release 0.3.2 published 2026-03-18, 84 contributors (distinct committers including anonymous), open_issues_count 342 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 21788 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Notes
Fourth of the structured-generation records (with [[niche-xgrammar]], [[niche-outlines]], [[niche-lm-format-enforcer]]) because it is the one with a genuinely different mechanism: it constrains and backtracks rather than only restricting logits. That mechanism is why it survives alongside the logit-restriction libraries rather than being displaced by them. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-05-21 (4.5 months before this record), latest release 0.3.2 published 2026-03-18, 84 contributors (distinct committers including anonymous), open_issues_count 342 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 21788 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Cross-references
This record references no other records — it is a leaf. It is cited by 0.