inference/research
verifiedmedium · 0.70updated 2026-10-04

Microsoft Olive (Onnx LIVE) - hardware-targeted ONNX optimization / quantization composer

not a single format: Olive runs search over conversion + quantization passes and emits whichever ONNX quant configuration wins for the declared target. Quantization algorithms were expanded in Olive 0.8.0 (2025-03) and again in 0.9.2 (2025-08)

Fields

id
quantization/olive-quant-stack
Scheme
int8
Bits
not a single format: Olive runs search over conversion + quantization passes and emits whichever ONNX quant configuration wins for the declared target. Quantization algorithms were expanded in Olive 0.8.0 (2025-03) and again in 0.9.2 (2025-08)
Activation scheme
target-dependent; NPU support arrived in Olive 0.9.0 (2025-05)
Kernels
Olive emits an ONNX graph + external data; the runtime kernels are ONNX Runtime's, not Olive'sper-tensor / per-channel quantization configs searched against a latency+accuracy constraint
Quality delta
n/a - no quality delta published. Null + note.
notes
PROXY ENUM VALUE - DECLARED 2026-10-04 (already partially self-declared, now explicit): Olive is a search orchestrator whose quant format is an OUTPUT of a hardware-targeted search, filed under 'int8'. The record deliberately declines to claim a group size or bit width because the format is not an input to the tool - but the scheme field still asserts INT8, which is the one thing the record explicitly refuses to claim. This is an inconsistency between the notes and the enum within a single record. REQUESTED ENUM ADDITION: 'olive'. GROUP SIZE LEFT NULL 2026-10-04 by design: Olive is a search orchestrator whose quant format is an OUTPUT of a hardware-targeted search, not an input. The record already states it deliberately declines to claim a group size or bit width. Nothing to read. EXISTS (MIT, ~2.4k stars, 2k+ commits) and is NOT a quantizer in the llama.cpp sense - it is a search orchestrator. Its own framing, quoted: 'Given a model and targeted hardware, Olive composes the best suitable optimization techniques to output the most efficient ONNX model(s) for inferencing on the cloud or edge, while taking a set of constraints such as accuracy and latency into consideration.' So the quant format is an OUTPUT of a search, not an input, and the record deliberately does not claim a group size or a bit width - null rather than a guess. WHY IT IS WORTH RECORDING even with those nulls: it is the Microsoft-side answer to 'which quant should I pick', it is the tool the ONNX Runtime ecosystem expects to sit between a PyTorch checkpoint and an ORT session, and it is where NPU targets enter the picture (Olive 0.9.0, May 2025). Its own quickstart installs `transformers onnxruntime-genai` together, and the docs ship an Olive blog specifically on exploring optimal quantization settings for small language models (Oct 2025). SCOPE CAVEATS: examples moved to a separate olive-recipes repository in Sep 2025; and the ONNX Runtime GenAI builder README explicitly lists 'Olive integration' as still PENDING, so the two do not yet compose. quantization_record is a null-scheme-leaning record by design. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: NOT A STORAGE FORMAT, SO THE FIELD IS DEFINITIONALLY EMPTY. This record describes a calibration file, a search orchestrator, removed enum slots or a cross-method summary rather than a wire format, and no accelerator can natively support any of those. Consequence for a reader: nothing further to verify - the question does not apply to this record. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

PROXY ENUM VALUE - DECLARED 2026-10-04 (already partially self-declared, now explicit): Olive is a search orchestrator whose quant format is an OUTPUT of a hardware-targeted search, filed under 'int8'. The record deliberately declines to claim a group size or bit width because the format is not an input to the tool - but the scheme field still asserts INT8, which is the one thing the record explicitly refuses to claim. This is an inconsistency between the notes and the enum within a single record. REQUESTED ENUM ADDITION: 'olive'. GROUP SIZE LEFT NULL 2026-10-04 by design: Olive is a search orchestrator whose quant format is an OUTPUT of a hardware-targeted search, not an input. The record already states it deliberately declines to claim a group size or bit width. Nothing to read. EXISTS (MIT, ~2.4k stars, 2k+ commits) and is NOT a quantizer in the llama.cpp sense - it is a search orchestrator. Its own framing, quoted: 'Given a model and targeted hardware, Olive composes the best suitable optimization techniques to output the most efficient ONNX model(s) for inferencing on the cloud or edge, while taking a set of constraints such as accuracy and latency into consideration.' So the quant format is an OUTPUT of a search, not an input, and the record deliberately does not claim a group size or a bit width - null rather than a guess. WHY IT IS WORTH RECORDING even with those nulls: it is the Microsoft-side answer to 'which quant should I pick', it is the tool the ONNX Runtime ecosystem expects to sit between a PyTorch checkpoint and an ORT session, and it is where NPU targets enter the picture (Olive 0.9.0, May 2025). Its own quickstart installs `transformers onnxruntime-genai` together, and the docs ship an Olive blog specifically on exploring optimal quantization settings for small language models (Oct 2025). SCOPE CAVEATS: examples moved to a separate olive-recipes repository in Sep 2025; and the ONNX Runtime GenAI builder README explicitly lists 'Olive integration' as still PENDING, so the two do not yet compose. quantization_record is a null-scheme-leaning record by design. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: NOT A STORAGE FORMAT, SO THE FIELD IS DEFINITIONALLY EMPTY. This record describes a calibration file, a search orchestrator, removed enum slots or a cross-method summary rather than a wire format, and no accelerator can natively support any of those. Consequence for a reader: nothing further to verify - the question does not apply to this record. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references