inference/research
verifiedhigh · 0.90updated 2026-10-04

llama.cpp tools/server/README.md: llama-server feature and flag surface (read 2026-10-04)

blog

Fields

id
sources/ev-llamacpp-server-readme-feature-axes
Publisher
llama.cpp
Kind
blog
Accessed
2026-10-04
notes
Read 2026-10-04 from ggml-org/llama.cpp master, used to fill the feature-comparison axes for [[llama-cpp]]. SPECULATIVE DECODING is broader than any other engine in this repo on the count of distinct methods: --spec-type accepts a comma-separated list of 'none,draft-simple,draft-eagle3,draft-mtp,draft-dflash,draft-dspark,ngram-simple,ngram-map-k,ngram-map-k4v,ngram-mod,ngram-cache' - eleven distinct algorithms across three families (small draft models incl. EAGLE3/MTP, n-gram lookup with five cache-keying variants, and dflash/dspark). Tuning knobs are explicit: --spec-draft-n-max (default 3), --spec-draft-n-min (default 0), --spec-draft-p-split (default 0.10), --spec-draft-p-min (default 0.00), and n-gram sizing --spec-ngram-simple-size-n/m (12/48), --spec-ngram-map-k-size-n/m (12/48), --spec-ngram-mod-n-min/n-max (48/64). STRUCTURED OUTPUT: --grammar (BNF-like, samples in grammars/), --grammar-file, -j/--json-schema, -jf/--json-schema-file. PREFIX/SLOT CACHE, which is llama.cpp's actual mechanism rather than a named prefix-caching feature: --kv-unified/-no-kv-unified 'use single unified KV buffer shared across all sequences (default: enabled if number of slots is auto)'; -sps/--slot-prompt-similarity 'how much the prompt of a request must match the prompt of a slot in order to use that slot (default: 0.10, 0.0 = disabled)'; --cache-idle-slots 'save idle slots to the prompt cache on new task, and clear them when using unified KV'; --slot-save-path; --kv-unified-per-slot; and -ctxcp/--ctx-checkpoints 'max number of context checkpoints to create per slot (default: 32)'. MULTIMODAL: -mm/--mmproj, --mmproj-url, --mmproj-auto/--no-mmproj-auto, --mmproj-offload, -mmdev/--mmproj-device, and dynamic-resolution controls --image-min-tokens / --image-max-tokens 'only used by vision models with dynamic resolution'; the model-table tool downloads mmproj automatically with --hf-repo unless --no-mmproj. SCHEDULING: -np/--parallel N server slots (default -1 = auto), -cb/--cont-batching 'whether to enable continuous batching (a.k.a dynamic batching) (default: enabled)' and its negation, --slots monitoring endpoint, and --pooling {none,mean,cls,last,rank} for embeddings with --embd-normalize N (default 2). NOTE what is ABSENT from this surface, which is the honest negative: there is no disaggregated prefill/decode mode in llama-server, and no chunked-prefill knob under that name - so for those two axes the answer for llama.cpp is 'no', not 'unknown'.

Notes

Read 2026-10-04 from ggml-org/llama.cpp master, used to fill the feature-comparison axes for [[llama-cpp]]. SPECULATIVE DECODING is broader than any other engine in this repo on the count of distinct methods: --spec-type accepts a comma-separated list of 'none,draft-simple,draft-eagle3,draft-mtp,draft-dflash,draft-dspark,ngram-simple,ngram-map-k,ngram-map-k4v,ngram-mod,ngram-cache' - eleven distinct algorithms across three families (small draft models incl. EAGLE3/MTP, n-gram lookup with five cache-keying variants, and dflash/dspark). Tuning knobs are explicit: --spec-draft-n-max (default 3), --spec-draft-n-min (default 0), --spec-draft-p-split (default 0.10), --spec-draft-p-min (default 0.00), and n-gram sizing --spec-ngram-simple-size-n/m (12/48), --spec-ngram-map-k-size-n/m (12/48), --spec-ngram-mod-n-min/n-max (48/64). STRUCTURED OUTPUT: --grammar (BNF-like, samples in grammars/), --grammar-file, -j/--json-schema, -jf/--json-schema-file. PREFIX/SLOT CACHE, which is llama.cpp's actual mechanism rather than a named prefix-caching feature: --kv-unified/-no-kv-unified 'use single unified KV buffer shared across all sequences (default: enabled if number of slots is auto)'; -sps/--slot-prompt-similarity 'how much the prompt of a request must match the prompt of a slot in order to use that slot (default: 0.10, 0.0 = disabled)'; --cache-idle-slots 'save idle slots to the prompt cache on new task, and clear them when using unified KV'; --slot-save-path; --kv-unified-per-slot; and -ctxcp/--ctx-checkpoints 'max number of context checkpoints to create per slot (default: 32)'. MULTIMODAL: -mm/--mmproj, --mmproj-url, --mmproj-auto/--no-mmproj-auto, --mmproj-offload, -mmdev/--mmproj-device, and dynamic-resolution controls --image-min-tokens / --image-max-tokens 'only used by vision models with dynamic resolution'; the model-table tool downloads mmproj automatically with --hf-repo unless --no-mmproj. SCHEDULING: -np/--parallel N server slots (default -1 = auto), -cb/--cont-batching 'whether to enable continuous batching (a.k.a dynamic batching) (default: enabled)' and its negation, --slots monitoring endpoint, and --pooling {none,mean,cls,last,rank} for embeddings with --embd-normalize N (default 2). NOTE what is ABSENT from this surface, which is the honest negative: there is no disaggregated prefill/decode mode in llama-server, and no chunked-prefill knob under that name - so for those two axes the answer for llama.cpp is 'no', not 'unknown'.

Cross-references

This record references no other records.