inference/research
verifiedhigh · 0.90updated 2026-10-03

ONNX Runtime INT4/UInt4 via MatMulNBits and GatherBlockQuantized (weight-only, blockwise)

4-bit weights (Int4 if is_symmetric=True, else UInt4). Block size must be a power of two >= 16; the documented example is block_size=128

Fields

id
quantization/ort-matmulnbits-int4
Scheme
ort-matmulnbits-int4
Bits
4-bit weights (Int4 if is_symmetric=True, else UInt4). Block size must be a power of two >= 16; the documented example is block_size=128
Weight group size
128
Activation scheme
none - weight-only. Activations stay fp32; the QDQ form is DequantizeLinear -> MatMul
Kernels
MatMulNBits (com.microsoft contrib op) - blockwise weight-only 4-bit GEMMGatherBlockQuantized (com.microsoft contrib op) - RTN only, Gather with constant dataMatMulNBits accuracy_level 1-4 attribute (4 = best, the documented example value)QDQ form: DequantizeLinear holding the blockwise-quantized initializer -> plain MatMul
Quality delta
n/a - no numeric delta published by Microsoft. Null + note.
notes
SCHEMA MISMATCH: there is no int4/uint4 enum value at all. 'gptq' is the closest existing value because MatMulNBits is weight-only blockwise 4-bit with per-block scales - the same shape as GPTQ - and GPTQ is one of the three algorithms ORT accepts for it. The true width and type are in `name` and `bits`. Requested enum addition: 'ort-matmulnbits-int4'. This is the INT4 half of the ORT toolset and it is the part with a real hardware path, unlike the 8-bit static/dynamic story. Supported ops: MatMul (quantized only when input B is constant) and Gather (only when input data is constant). ALGORITHMS: for MatMulNBits the tool supports HQQ, GPTQ and RTN - RTN being the DEFAULT when nothing else is specified. For GatherBlockQuantized only RTN is supported. So HQQ is reachable from ONNX Runtime as a MatMulNBits producer, which is a genuinely useful cross-ecosystem fact and the only place HQQ has a first-class ORT integration. OPERATIONAL CONSTRAINTS, all documented: Int4/UInt4 were introduced in ONNX opset 21, so a model on an older opset is force-upgraded to 21 and the operators must be opset-21-compatible; running a model containing GatherBlockQuantized requires ONNX Runtime 1.20 or later; `is_symmetric=True` selects Int4 and False selects UInt4; quant_axes is per-op-type (the example uses ("MatMul", 0) and ("Gather", 1)); and accuracy_level is an attribute on MatMulNBits with 4 as the documented value. AWQ and GPTQ are deliberately NOT produced by this API - the docs redirect to the ONNX Runtime GenAI model builder for those. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'gptq' (a proxy recorded while the enum lacked a legal value) and is now 'ort-matmulnbits-int4'.

Notes

SCHEMA MISMATCH: there is no int4/uint4 enum value at all. 'gptq' is the closest existing value because MatMulNBits is weight-only blockwise 4-bit with per-block scales - the same shape as GPTQ - and GPTQ is one of the three algorithms ORT accepts for it. The true width and type are in `name` and `bits`. Requested enum addition: 'ort-matmulnbits-int4'. This is the INT4 half of the ORT toolset and it is the part with a real hardware path, unlike the 8-bit static/dynamic story. Supported ops: MatMul (quantized only when input B is constant) and Gather (only when input data is constant). ALGORITHMS: for MatMulNBits the tool supports HQQ, GPTQ and RTN - RTN being the DEFAULT when nothing else is specified. For GatherBlockQuantized only RTN is supported. So HQQ is reachable from ONNX Runtime as a MatMulNBits producer, which is a genuinely useful cross-ecosystem fact and the only place HQQ has a first-class ORT integration. OPERATIONAL CONSTRAINTS, all documented: Int4/UInt4 were introduced in ONNX opset 21, so a model on an older opset is force-upgraded to 21 and the operators must be opset-21-compatible; running a model containing GatherBlockQuantized requires ONNX Runtime 1.20 or later; `is_symmetric=True` selects Int4 and False selects UInt4; quant_axes is per-op-type (the example uses ("MatMul", 0) and ("Gather", 1)); and accuracy_level is an attribute on MatMulNBits with 4 as the documented value. AWQ and GPTQ are deliberately NOT produced by this API - the docs redirect to the ONNX Runtime GenAI model builder for those. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'gptq' (a proxy recorded while the enum lacked a legal value) and is now 'ort-matmulnbits-int4'.

Cross-references

Native support (10)