ONNX Runtime quantization toolset (quantize_static / quantize_dynamic, QDQ vs QOperator)
Fields
quantization/ort-quant-int8Notes
SCHEMA MISMATCH: the enum has 'int8' but the sibling record is filed under id int8-w8a8 and the enum value 'int8-w8a8' does not exist; 'int8' is the closest existing value and the true scope is in `name`. THE CPU-vs-GPU-vs-NPU MATRIX, which the docs genuinely do make confusing, resolved. CPU (MLAS): runs U8U8, U8S8 and S8S8. S8S8 with QDQ is the documented default and the first choice; S8S8 with QOperator is explicitly documented as SLOW on x86-64 and to be avoided; U8U8 is the fallback when accuracy drops badly. GPU: S8S8 ONLY. NPU (NNAPI on Android): QLinearConv and QLinearMatMul with all quantization scales and zero points constant - so an NNAPI path needs static, constant scales, which rules out dynamic activation quantization on NPU. QDQ vs QOperator: QOperator gives each quantized op its own ONNX definition (QLinearConv, MatMulInteger); QDQ inserts DeQuantizeLinear(QuantizeLinear(x)) around the original ops and carries the parameters on those ops - in static mode in the graph, in dynamic mode via an inserted ComputeQuantizationParameters proto. QDQ has been the DEFAULT since ORT 1.11 and ORT's own CPU kernels are optimized for it; QOperator is described as 'mainly useful for specific hardware back-ends'. PER-PLATFORM PARAMETER TABLE (the part people get wrong): x86 WITHOUT VNNI -> activation QUInt8 + weight QInt8 + reduce_range=True (7-bit weights) to avoid VPMADDUBSW 16-bit saturation; x86 WITH VNNI (Skylake-SP/Cascade Lake/Ice Lake/Sapphire Rapids, AMD Zen4) -> same types but reduce_range=False, because VNNI accumulates 8-bit products without saturating; ARM (Cortex-A, Apple Silicon, Graviton) -> QInt8/QInt8, reduce_range=False, because NEON/SVE handles signed 8-bit without saturation. per_channel=False gives better CPU throughput; per_channel=True can help models with large inter-channel weight ranges but on AVX2/AVX512 you generally need reduce_range as well. The docs state plainly that choosing the wrong combination is 'a common cause of quantized models running slower than FP32'. HARDWARE PRECONDITION, quoted: 'Old hardware has none or few of the instructions needed to perform efficient inference in int8. And quantization has overhead (from quantizing and dequantizing), so it is not rare to get worse performance on old devices.' Only x86-64 with VNNI, GPU with Tensor Core int8 support, and ARM with dot-product instructions reliably win. OPERATIONAL GOTCHAS: models must be opset >= 10 (opset >= 12 for 8-bit MaxPool); model optimization cannot output a model larger than 2GB, so large models must skip it, and the docs recommend running optimization during pre-processing (quant_pre_process) rather than during quantization, precisely because in-quantization optimization rewrites the graph and destroys tensor-to-tensor matching for the debugging APIs. quantize_static supports three calibration methods - MinMax, Entropy, Percentile.