inference/research
verifiedhigh · 0.85minorupdated 2026-10-04

The seven feature axes a reader chooses an engine on were recorded for a minority of engines - a dated audit of which are stated, which are absent, and which are affirmatively NO

[minor] measurement

Fields

id
gotchas/ev-feature-axis-coverage-audit
Class
measurement
Severity
minor
Concepts
feature-comparisonspeculative-decodingchunked-prefillprefix-cachingdisaggregated-servingstructured-outputmultimodalmulti-modelkv-cachenegative-results
Symptom
A reader comparing engines on 'does it do speculative decoding, chunked prefill, prefix caching, disaggregated prefill/decode, structured output, multi-model, multimodal' finds these recorded for some engines and simply missing for others, with no way to tell 'lacks it' from 'nobody checked'. Audited on 2026-10-04 across all 103 engine records by scanning each record's serialized text for axis-specific vocabulary, then reading primary docs for the engines that mattered. RESULT: of the 103 engine records, 29 recorded none of the seven axes at all, and a further 15 recorded exactly one. Only 6 engines recorded three or more.
Root cause
The axes were recorded opportunistically - when an engine's author happened to mention speculative decoding in a README, that got written into notable_features, and nobody went looking for the axes that were not mentioned. The failure mode is silent and it is the dangerous kind: an absent field in a comparison table reads as 'no' to a hurried reader, which is wrong for an engine that supports the feature and simply was never checked. This audit exists so that an absent field reads as 'unstated' instead.
Workaround
For the seven axes, treat the only three legitimate values as YES (with a source), NO (affirmatively documented as absent, with a source), and UNSTATED (nobody checked - the default for a gap). Never infer NO from silence. Where this audit found a genuine YES it is named below with its source; where it found an affirmative NO it is named as such; everything else is UNSTATED and should stay that way until someone reads the engine's own docs. Concretely, the gaps worth closing next, in the order a reader is most likely to hit them: engines serving production traffic whose axis status is entirely UNSTATED ([[triton-inference-server]], [[niche-ray-serve]], [[niche-kserve]], [[nvidia-dynamo]], [[llm-d]] - all four axes are unstated for all of these), the consumer-local engines ([[ollama]], [[jan]], [[koboldcpp]], [[llamafile]], [[mlc-llm]]), and the ASR/image engines where 'speculative decoding' does not even apply but 'chunked prefill' and 'prefix caching' might ([[faster-whisper]], [[whisper-cpp]], [[openai-whisper]], [[apple-coreml]]).
notes
WHAT THE AUDIT FOUND, stated per axis as of 2026-10-04. SPECULATIVE DECODING - confirmed YES from vendor docs on 2026-10-04 for [[vllm]] (thirteen published methods: EAGLE, LiLiCorr, MTP, Draft Model, PARD, MLP speculator, N-gram, Suffix decoding, Hidden State Extraction, Custom Proposer Backend, Dynamic Speculative Decoding, Adaptive Verification, Per-Request Acceptance Metrics), [[sglang]] (EAGLE-2/EAGLE-3, MTP, UNO, DFLASH, classic draft model, NGRAM), [[llama-cpp]] (eleven algorithms across three families: draft-simple, draft-eagle3, draft-mtp, draft-dflash, draft-dspark, ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache), [[exllamav3]] (vendor README states it), [[openvino-genai]] (already recorded). UNSTATED for the rest. Two records that exist in this repo cover the technique as a system rather than as an engine feature and are not engine-axis answers: [[niche-eagle]] and [[niche-medusa]]. CHUNKED PREFILL - confirmed YES only for [[vllm]], [[sglang]] and [[vllm-gaudi]] (already recorded); UNSTATED for every other engine including [[tensorrt-llm]] and [[text-generation-inference]], which is worth noting because chunked prefill is a scheduling behaviour a reader would assume any modern engine has. PREFIX CACHING - the widest-covered axis and the one where the mechanisms genuinely differ, which is why a boolean would lose information: [[sglang]] radix tree over the KV cache (RadixAttention, any shared prefix not just an exact match); [[ktransformers]] three-tier GPU-CPU-DISK prefix cache reuse, the only engine here with a disk tier, and unsurprising once you know its weights do not fit in VRAM; [[openvino-genai]] caches 'fragments of previous generation requests'; [[llama-cpp]] has no named feature but the mechanisms underneath - a shared unified KV buffer, slot-prompt-similarity matching at default 0.10, idle-slot prompt-cache persistence, and up to 32 per-slot context checkpoints. DISAGGREGATED PREFILL/DECODE - YES on [[tensorrt-llm]] and [[vllm]] already; [[niche-lmdeploy-turbomind]] gains a qualified YES ('[2025/06] DeepSeek PD Disaggregation deployment is now supported through integration with DLSlime and Mooncake' - i.e. via two OTHER projects' integration, not an in-tree scheduler, the same distinction applied to vLLM's third-party connectors). vLLM's docs also contribute the caveat worth carrying forward: 'Disaggregated prefill DOES NOT improve throughput'. STRUCTURED OUTPUT - [[vllm]] (xgrammar and guidance backends, on by default in the OpenAI-compatible server, with an API break: guided_json/guided_regex/guided_choice/guided_grammar/guided_whitespace_pattern/structural_tag/guided_decoding_backend all REMOVED in v0.12.0) and [[onnxruntime-genai]] (grammar specification for tool calling is an in-engine first-class capability, unusual at that EPU breadth). UNSTATED elsewhere, including [[tensorrt-llm]]. MULTI-MODEL - the axis most often confused with multi-SURFACE: [[vllm]] and [[text-generation-inference]] (Rust router) already recorded; [[niche-lmdeploy-turbomind]] confirmed via a proxy/distribution server doc. [[mlc-llm]]'s existing 'One MLCEngine behind Python, REST, JavaScript/WebGPU, iOS and Android' is about serving SURFACES, not about serving several models at once, and the audit flags that as a distinction to preserve rather than an axis answer. MULTIMODAL - confirmed YES for [[exllamav3]], [[openvino-genai]] (image AND video, LLaVa/MiniCPM-V), [[ktransformers]] (GLM-5.3-flash, 2026-08-26), [[onnxruntime-genai]] (stated per-architecture, which is the more honest form: LFM2 language+vision+speech, Phi language+vision, Qwen language+vision). [[llama-cpp]] gains the most detailed answer in the repo from tools/server/README.md: --mmproj, --mmproj-url, --mmproj-auto, --mmproj-offload, --mmproj-device, --image-min-tokens/--image-max-tokens for dynamic-resolution models. NEGATIVE FINDINGS THAT ARE REAL NEGATIVES, not gaps: llama.cpp's server surface has NO disaggregated prefill/decode mode and no chunked-prefill knob, and no multimodal/audio projector path beyond the mmproj vision flags above. AFFIRMATIVE NO for vLLM multimodal on AMD: the existing [[vllm]] record already records that the enc-dec feature is unsupported on ROCm, which is a per-platform NO and a different claim from a general one. UNVERIFIED BY ME: [[mlc-llm]] is a stated-as-UNSTATED rather than a stated-as-NO - a keyword scan of its README returned zero hits on all seven axes, but the README is a short project overview and the authoritative per-model surface (the MLC model library) was NOT enumerated, so absence of evidence there is not evidence of absence; [[sglang]] chunked prefill and [[tensorrt-llm]] structured output were confirmed only from docs read on 2026-10-04 and should be re-checked against a specific release before being quoted as release-specific facts; and no axis was checked for [[fastertransformer]] beyond what its existing record already says.

Notes

WHAT THE AUDIT FOUND, stated per axis as of 2026-10-04. SPECULATIVE DECODING - confirmed YES from vendor docs on 2026-10-04 for [[vllm]] (thirteen published methods: EAGLE, LiLiCorr, MTP, Draft Model, PARD, MLP speculator, N-gram, Suffix decoding, Hidden State Extraction, Custom Proposer Backend, Dynamic Speculative Decoding, Adaptive Verification, Per-Request Acceptance Metrics), [[sglang]] (EAGLE-2/EAGLE-3, MTP, UNO, DFLASH, classic draft model, NGRAM), [[llama-cpp]] (eleven algorithms across three families: draft-simple, draft-eagle3, draft-mtp, draft-dflash, draft-dspark, ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache), [[exllamav3]] (vendor README states it), [[openvino-genai]] (already recorded). UNSTATED for the rest. Two records that exist in this repo cover the technique as a system rather than as an engine feature and are not engine-axis answers: [[niche-eagle]] and [[niche-medusa]]. CHUNKED PREFILL - confirmed YES only for [[vllm]], [[sglang]] and [[vllm-gaudi]] (already recorded); UNSTATED for every other engine including [[tensorrt-llm]] and [[text-generation-inference]], which is worth noting because chunked prefill is a scheduling behaviour a reader would assume any modern engine has. PREFIX CACHING - the widest-covered axis and the one where the mechanisms genuinely differ, which is why a boolean would lose information: [[sglang]] radix tree over the KV cache (RadixAttention, any shared prefix not just an exact match); [[ktransformers]] three-tier GPU-CPU-DISK prefix cache reuse, the only engine here with a disk tier, and unsurprising once you know its weights do not fit in VRAM; [[openvino-genai]] caches 'fragments of previous generation requests'; [[llama-cpp]] has no named feature but the mechanisms underneath - a shared unified KV buffer, slot-prompt-similarity matching at default 0.10, idle-slot prompt-cache persistence, and up to 32 per-slot context checkpoints. DISAGGREGATED PREFILL/DECODE - YES on [[tensorrt-llm]] and [[vllm]] already; [[niche-lmdeploy-turbomind]] gains a qualified YES ('[2025/06] DeepSeek PD Disaggregation deployment is now supported through integration with DLSlime and Mooncake' - i.e. via two OTHER projects' integration, not an in-tree scheduler, the same distinction applied to vLLM's third-party connectors). vLLM's docs also contribute the caveat worth carrying forward: 'Disaggregated prefill DOES NOT improve throughput'. STRUCTURED OUTPUT - [[vllm]] (xgrammar and guidance backends, on by default in the OpenAI-compatible server, with an API break: guided_json/guided_regex/guided_choice/guided_grammar/guided_whitespace_pattern/structural_tag/guided_decoding_backend all REMOVED in v0.12.0) and [[onnxruntime-genai]] (grammar specification for tool calling is an in-engine first-class capability, unusual at that EPU breadth). UNSTATED elsewhere, including [[tensorrt-llm]]. MULTI-MODEL - the axis most often confused with multi-SURFACE: [[vllm]] and [[text-generation-inference]] (Rust router) already recorded; [[niche-lmdeploy-turbomind]] confirmed via a proxy/distribution server doc. [[mlc-llm]]'s existing 'One MLCEngine behind Python, REST, JavaScript/WebGPU, iOS and Android' is about serving SURFACES, not about serving several models at once, and the audit flags that as a distinction to preserve rather than an axis answer. MULTIMODAL - confirmed YES for [[exllamav3]], [[openvino-genai]] (image AND video, LLaVa/MiniCPM-V), [[ktransformers]] (GLM-5.3-flash, 2026-08-26), [[onnxruntime-genai]] (stated per-architecture, which is the more honest form: LFM2 language+vision+speech, Phi language+vision, Qwen language+vision). [[llama-cpp]] gains the most detailed answer in the repo from tools/server/README.md: --mmproj, --mmproj-url, --mmproj-auto, --mmproj-offload, --mmproj-device, --image-min-tokens/--image-max-tokens for dynamic-resolution models. NEGATIVE FINDINGS THAT ARE REAL NEGATIVES, not gaps: llama.cpp's server surface has NO disaggregated prefill/decode mode and no chunked-prefill knob, and no multimodal/audio projector path beyond the mmproj vision flags above. AFFIRMATIVE NO for vLLM multimodal on AMD: the existing [[vllm]] record already records that the enc-dec feature is unsupported on ROCm, which is a per-platform NO and a different claim from a general one. UNVERIFIED BY ME: [[mlc-llm]] is a stated-as-UNSTATED rather than a stated-as-NO - a keyword scan of its README returned zero hits on all seven axes, but the README is a short project overview and the authoritative per-model surface (the MLC model library) was NOT enumerated, so absence of evidence there is not evidence of absence; [[sglang]] chunked prefill and [[tensorrt-llm]] structured output were confirmed only from docs read on 2026-10-04 and should be re-checked against a specific release before being quoted as release-specific facts; and no axis was checked for [[fastertransformer]] beyond what its existing record already says.

Cross-references

Affects (28)