inference/research
verifiedhigh · 0.90updated 2026-10-04

onnxruntime-genai README: generative-loop components, model architectures and EPU coverage (read 2026-10-04)

blog

Fields

id
sources/ev-onnxruntime-genai-readme-feature-axes
Publisher
Microsoft
Kind
blog
Accessed
2026-10-04
notes
Read 2026-10-04, used to fill the feature-comparison axes for [[onnxruntime-genai]]. STRUCTURED OUTPUT is the axis this record gains most: the README states the API implements 'pre and post processing, inference with ONNX Runtime, logits processing, search and sampling, KV cache management, and GRAMMAR SPECIFICATION FOR TOOL CALLING' - so grammar-constrained tool calling is an in-engine first-class capability rather than something bolted on by a companion library, which is unusual at this EPU breadth. MULTIMODAL is stated per-architecture in the supported table rather than as a blanket claim, which is the more honest form: LFM2 '(language + vision + speech input)', Phi '(language + vision)', Qwen '(language + vision)', Whisper, plus a separate 'Multi-modal models' column and 'Stable diffusion' coverage. Also notable for a reader choosing on axes: the listed architectures include DeepSeek, Nemotron, gpt-oss, Granite MoE Hybrid and Fara, so the model menu is broader than the record previously implied. PREFIX CACHING / CHUNKED PREFILL / DISAGGREGATED P/D / SPECULATIVE DECODING: none of the four appears in the README, so all four remain UNSTATED for this engine. The record's own 'scheduling: none' stands, with the qualification that KV cache management is present but is not scheduling.

Notes

Read 2026-10-04, used to fill the feature-comparison axes for [[onnxruntime-genai]]. STRUCTURED OUTPUT is the axis this record gains most: the README states the API implements 'pre and post processing, inference with ONNX Runtime, logits processing, search and sampling, KV cache management, and GRAMMAR SPECIFICATION FOR TOOL CALLING' - so grammar-constrained tool calling is an in-engine first-class capability rather than something bolted on by a companion library, which is unusual at this EPU breadth. MULTIMODAL is stated per-architecture in the supported table rather than as a blanket claim, which is the more honest form: LFM2 '(language + vision + speech input)', Phi '(language + vision)', Qwen '(language + vision)', Whisper, plus a separate 'Multi-modal models' column and 'Stable diffusion' coverage. Also notable for a reader choosing on axes: the listed architectures include DeepSeek, Nemotron, gpt-oss, Granite MoE Hybrid and Fara, so the model menu is broader than the record previously implied. PREFIX CACHING / CHUNKED PREFILL / DISAGGREGATED P/D / SPECULATIVE DECODING: none of the four appears in the README, so all four remain UNSTATED for this engine. The record's own 'scheduling: none' stands, with the qualification that KV cache management is present but is not scheduling.

Cross-references

This record references no other records.