inference/research
verifiedhigh · 0.85updated 2026-10-03

llama.cpp src/llama-grammar.cpp (master) - per-token grammar acceptance on the host

repo

Fields

id
sources/flop3-llamacpp-grammar-source
Publisher
ggml-org/llama.cpp
Kind
repo
Accessed
2026-10-03
notes
The host-side incremental grammar loop. llama_grammar_accept_impl fetches grammar.vocab->token_to_piece(token) for EVERY accepted token, then llama_grammar_accept_token decodes UTF-8 and, for each code point, copies the live parse stack per stack (llama_grammar_stack new_stack(stack.begin(), stack.end() - 1)) as it advances the state machine. Trigger handling keeps a growing trigger_buffer plus a trigger_buffer_positions vector of (token, byte-range) pairs and re-runs each trigger_pattern.find() against the accumulated buffer every token. So the per-token cost is (token-to-piece lookup) + (bytes of the piece) + (stack copies) + (regex finds per trigger pattern) - all on the CPU, once per generated token, and it grows with the number of stacks and the trigger buffer length.

Notes

The host-side incremental grammar loop. llama_grammar_accept_impl fetches grammar.vocab->token_to_piece(token) for EVERY accepted token, then llama_grammar_accept_token decodes UTF-8 and, for each code point, copies the live parse stack per stack (llama_grammar_stack new_stack(stack.begin(), stack.end() - 1)) as it advances the state machine. Trigger handling keeps a growing trigger_buffer plus a trigger_buffer_positions vector of (token, byte-range) pairs and re-runs each trigger_pattern.find() against the accumulated buffer every token. So the per-token cost is (token-to-piece lookup) + (bytes of the piece) + (stack copies) + (regex finds per trigger pattern) - all on the CPU, once per generated token, and it grows with the number of stacks and the trigger buffer length.

Cross-references

This record references no other records.