inference/research
verifiedmedium · 0.75updated 2026-10-04

LLM.int8() / bitsandbytes int8 (vector-wise with 16-bit outliers)

w8a16 (mixed-precision decomposition: >99.9% of values multiplied in 8-bit, outlier dims in fp16)

Fields

id
quantization/bitsandbytes-int8
Scheme
bitsandbytes-int8
Bits
w8a16 (mixed-precision decomposition: >99.9% of values multiplied in 8-bit, outlier dims in fp16)
Activation scheme
Vector-wise quantization: SEPARATE NORMALIZATION CONSTANTS FOR EACH INNER PRODUCT in the matrix multiplication. The second half of the scheme is the MIXED-PRECISION DECOMPOSITION: systematically-emergent outlier FEATURE DIMENSIONS (not outlier values, not outlier tokens - specific dimensions of the activation vector) are isolated into a second 16-bit matrix multiplication, while more than 99.9% of values are still multiplied in 8-bit. The outliers are discovered at runtime per matmul, so the split is dynamic, not a static per-layer decision.
Kernels
bitsandbytes Linear8bitLtLLM.int8() vector-wise matmul + mixed-precision decomposition kernel
Quality delta
From the primary paper (qlab-llmint8-paper): a 175B-parameter 16/32-bit checkpoint can be loaded, converted to Int8 and used immediately with no performance degradation, because the mixed-precision decomposition keeps 'more than 99.9% of values multiplied in 8-bit' while the outlier dimensions go through fp16. The authors frame the entire result as 'it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation'. No numeric benchmark delta is tabulated in the abstract, so no perplexity or accuracy number is recorded here.
notes
GROUP SIZE LEFT NULL 2026-10-04 and this is a genuine per-vector decomposition, not a missing value: LLM.int8() splits each matrix column into outlier columns (magnitude above a 6.0 threshold, kept in FP16) and non-outlier columns quantized per VECTOR at INT8. The unit is the column, which is not a group size - the whole tensor can have a different outlier count in every column. Do not fill 64 here: that is the NF4 blocksize from the 4-bit path and is a different scheme. The important structural point: LLM.int8() is NOT W8A8. It is vector-wise 8-bit weights AND 8-bit activations, but the systematic outlier channels are split out into a second fp16 matrix multiply, so there is no true int8 x int8 tensor-core op running for the whole layer. That decomposition is what makes it work at all; it also makes it slow - it was a memory-saving technique, not a speed technique. Contrast with SmoothQuant, which genuinely reaches hardware W8A8 by moving difficulty into the weights upfront. Group size null: constants are per vector, not per fixed-size weight group. UPGRADE (2026-10-03): activation_scheme and quality_delta now cite the primary paper (arXiv 2208.07339, read directly) rather than relying on a secondary description. The precise wording matters and is now correct: LLM.int8() isolates outlier FEATURE DIMENSIONS into a 16-bit matmul while >99.9% of values remain 8-bit - that is the mixed-precision decomposition that the method is named for. Why it is not a true int8xint8 path: the layer's arithmetic is split between an int8 GEMM and an fp16 GEMM, so no single int8 tensor-core instruction covers the layer. That is the same trap as W4A4-without-hardware: low precision in the DATA does not imply low precision in the ARITHMETIC. The family resemblance worth keeping explicit - LLM.int8()'s mixed-precision decomposition, Atom's mixed-precision block-outliers, SqueezeLLM's dense-and-sparse split, and KVQuant's per-vector dense-and-sparse are four papers independently arriving at 'keep the outliers in higher precision'. QuaRot is the counterargument to that family: rotate them away instead, and retain nothing. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE LEFT NULL 2026-10-04 and this is a genuine per-vector decomposition, not a missing value: LLM.int8() splits each matrix column into outlier columns (magnitude above a 6.0 threshold, kept in FP16) and non-outlier columns quantized per VECTOR at INT8. The unit is the column, which is not a group size - the whole tensor can have a different outlier count in every column. Do not fill 64 here: that is the NF4 blocksize from the 4-bit path and is a different scheme. The important structural point: LLM.int8() is NOT W8A8. It is vector-wise 8-bit weights AND 8-bit activations, but the systematic outlier channels are split out into a second fp16 matrix multiply, so there is no true int8 x int8 tensor-core op running for the whole layer. That decomposition is what makes it work at all; it also makes it slow - it was a memory-saving technique, not a speed technique. Contrast with SmoothQuant, which genuinely reaches hardware W8A8 by moving difficulty into the weights upfront. Group size null: constants are per vector, not per fixed-size weight group. UPGRADE (2026-10-03): activation_scheme and quality_delta now cite the primary paper (arXiv 2208.07339, read directly) rather than relying on a secondary description. The precise wording matters and is now correct: LLM.int8() isolates outlier FEATURE DIMENSIONS into a 16-bit matmul while >99.9% of values remain 8-bit - that is the mixed-precision decomposition that the method is named for. Why it is not a true int8xint8 path: the layer's arithmetic is split between an int8 GEMM and an fp16 GEMM, so no single int8 tensor-core instruction covers the layer. That is the same trap as W4A4-without-hardware: low precision in the DATA does not imply low precision in the ARITHMETIC. The family resemblance worth keeping explicit - LLM.int8()'s mixed-precision decomposition, Atom's mixed-precision block-outliers, SqueezeLLM's dense-and-sparse split, and KVQuant's per-vector dense-and-sparse are four papers independently arriving at 'keep the outliers in higher precision'. QuaRot is the counterargument to that family: rotate them away instead, and retain nothing. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)