inference/research
verifiedmedium · 0.65updated 2026-10-04

optimum-quanto (Hugging Face quantization library)

int2 / int4 / int8 / float8 weights (and optional int8/float8 activations)

Fields

id
quantization/optimum-quanto
Scheme
optimum-quanto
Bits
int2 / int4 / int8 / float8 weights (and optional int8/float8 activations)
Activation scheme
optional int8 or float8 activations; when only weights are quantized the runtime is 16-bit compute, when activations are quantized too it becomes a true low-precision matmul
Kernels
optimum-quanto CUDA kernels: int8-int8, fp16-int4, bf16-int8, bf16-int4
Quality delta
Qualitative only, from the project README: models compiled with int8/float8 weights and float8 activations are 'very close to the full-precision models', and inference latency with quantized-weights-only is comparable to full precision where optimized kernels exist (quant-quanto-readme). No numeric benchmark delta published - hence quality_delta left as this qualitative statement rather than null only.
notes
GROUP SIZE LEFT NULL 2026-10-04 and the reason is a real axis ambiguity: quanto's projection rule differs BY BITWIDTH, not just by tensor. int8 and float8 are symmetric per-tensor or per-channel; LOWER BITWIDTH (int4, int2) is GROUP-WISE AFFINE with a shift/zero-point. So 'the' group size depends on which dtype you mean, and quanto also lets the user choose per-channel along the FIRST axis (output features). One number cannot cover this. Notable for one specific claim: optimum-quanto provides an accelerated int8-int8 matmul, i.e. a real low-precision x low-precision kernel rather than dequant-to-fp16 - so quanto W8A8 (weights AND activations quantized) sits on the same side of the w4a16/w8a8 line as SmoothQuant, and quanto int4 weights alone sit on the other. The README's own summarization of its tradeoff: latency is comparable to full precision only when quantizing weights alone, and device memory divides roughly by float_bits/int_bits. It also supports int2, which is well outside what the tensor cores can consume. Unlike ModelOpt this library is hardware-agnostic - the int8-int8 kernel is a CUDA kernel, and nothing in it makes use of a tensor-core INT8 opcode being absent elsewhere. DTYPE AND AXIS DETAIL, added from the upstream design-overview README (qlab2-optimum-quanto-readme): the projection rule differs BY BITWIDTH, not just by tensor - it is symmetric per-tensor or per-channel for int8 and float8, and GROUP-WISE AFFINE (with a shift / zero-point) for lower bitwidth, which is why int4 and int2 carry real overhead. Weights are usually quantized per-channel along the FIRST dimension (output features), and weights inside matrix multiplications are ALWAYS quantized along that first axis because all output features of one accumulator share it - so 'per-channel' in quanto means per output feature, not per input channel, and a user assuming GPTQ/AWQ axis conventions will pick the wrong axis. Activations quantized per-tensor to int8 can fail badly when the tensor contains large outlier values, which the README says is a reason to prefer float8 activations. The library does not support mixed destination types within one tensor. Quantized modules convert weights dynamically until the model is frozen, which the README notes slightly slows inference but is required if the model is later tuned. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE LEFT NULL 2026-10-04 and the reason is a real axis ambiguity: quanto's projection rule differs BY BITWIDTH, not just by tensor. int8 and float8 are symmetric per-tensor or per-channel; LOWER BITWIDTH (int4, int2) is GROUP-WISE AFFINE with a shift/zero-point. So 'the' group size depends on which dtype you mean, and quanto also lets the user choose per-channel along the FIRST axis (output features). One number cannot cover this. Notable for one specific claim: optimum-quanto provides an accelerated int8-int8 matmul, i.e. a real low-precision x low-precision kernel rather than dequant-to-fp16 - so quanto W8A8 (weights AND activations quantized) sits on the same side of the w4a16/w8a8 line as SmoothQuant, and quanto int4 weights alone sit on the other. The README's own summarization of its tradeoff: latency is comparable to full precision only when quantizing weights alone, and device memory divides roughly by float_bits/int_bits. It also supports int2, which is well outside what the tensor cores can consume. Unlike ModelOpt this library is hardware-agnostic - the int8-int8 kernel is a CUDA kernel, and nothing in it makes use of a tensor-core INT8 opcode being absent elsewhere. DTYPE AND AXIS DETAIL, added from the upstream design-overview README (qlab2-optimum-quanto-readme): the projection rule differs BY BITWIDTH, not just by tensor - it is symmetric per-tensor or per-channel for int8 and float8, and GROUP-WISE AFFINE (with a shift / zero-point) for lower bitwidth, which is why int4 and int2 carry real overhead. Weights are usually quantized per-channel along the FIRST dimension (output features), and weights inside matrix multiplications are ALWAYS quantized along that first axis because all output features of one accumulator share it - so 'per-channel' in quanto means per output feature, not per input channel, and a user assuming GPTQ/AWQ axis conventions will pick the wrong axis. Activations quantized per-tensor to int8 can fail badly when the tensor contains large outlier values, which the README says is a reason to prefer float8 activations. The library does not support mixed destination types within one tensor. Quantized modules convert weights dynamically until the model is frozen, which the README notes slightly slows inference but is required if the model is later tuned. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (4)