inference/research
verifiedmedium · 0.80updated 2026-10-04

SqueezeLLM (dense-and-sparse quantization, up to 3-bit)

3-bit weights (paper claims lossless compression to ultra-low precisions of up to 3-bit); activations fp16

Fields

id
quantization/squeezellm
Scheme
squeezellm
Bits
3-bit weights (paper claims lossless compression to ultra-low precisions of up to 3-bit); activations fp16
Activation scheme
None - weight-only. Two components: (i) sensitivity-based non-uniform quantization that searches for the optimal bit-precision assignment using second-order information; (ii) a dense-and-sparse decomposition that stores outliers and sensitive weight values in a sparse format rather than spending a uniform grid on them.
Kernels
The paper frames the whole problem as memory-bandwidth-bound at single-batch, not compute-bound - the same framing that later becomes the "weight-only is a memory optimization" argument.
Quality delta
From the abstract: on LLaMA models, 3-bit quantization 'significantly reduces the perplexity gap from the FP16 baseline by up to 2.1x as compared to the state-of-the-art methods with the same memory' budget. The 2.1x is a RATIO OF THE GAP, not an absolute perplexity; do not read it as 'PPL is 2.1x worse'.
notes
GROUP SIZE LEFT NULL 2026-10-04 deliberately. LEFT NULL 2026-10-04 deliberately. SqueezeLLM's own comparison structure is AGAINST grouping: the paper compares 'dense-only SqueezeLLM with non-grouped GPTQ' and varies the group size of the GPTQ/AWQ baselines instead. The sparsity level (45% headline) is the tuning knob, not a group size. Setting 128 here would silently import GPTQ's convention into a method whose contribution is being non-grouped plus sparse. This is the paper that named the thing most blog coverage gets wrong: the abstract's opening claim is that the bottleneck for generative LLM inference is MEMORY BANDWIDTH, not compute, at single batch. That is precisely the w4a16 argument - you shrink weights to move fewer bytes, compute stays fp16. SqueezeLLM is therefore in the same regime as GPTQ/AWQ/EXL2, NOT in the W4A4 regime. Its dense-and-sparse outlier handling is the same idea as LLM.int8()'s mixed-precision decomposition and Atom's block-outlier scheme, applied to weights rather than activations. Enum mismatch: 'hqq' is nearest because both search for non-uniform quantization parameters from second-order information. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'hqq' (a proxy recorded while the enum lacked a legal value) and is now 'squeezellm'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE LEFT NULL 2026-10-04 deliberately. LEFT NULL 2026-10-04 deliberately. SqueezeLLM's own comparison structure is AGAINST grouping: the paper compares 'dense-only SqueezeLLM with non-grouped GPTQ' and varies the group size of the GPTQ/AWQ baselines instead. The sparsity level (45% headline) is the tuning knob, not a group size. Setting 128 here would silently import GPTQ's convention into a method whose contribution is being non-grouped plus sparse. This is the paper that named the thing most blog coverage gets wrong: the abstract's opening claim is that the bottleneck for generative LLM inference is MEMORY BANDWIDTH, not compute, at single batch. That is precisely the w4a16 argument - you shrink weights to move fewer bytes, compute stays fp16. SqueezeLLM is therefore in the same regime as GPTQ/AWQ/EXL2, NOT in the W4A4 regime. Its dense-and-sparse outlier handling is the same idea as LLM.int8()'s mixed-precision decomposition and Atom's block-outlier scheme, applied to weights rather than activations. Enum mismatch: 'hqq' is nearest because both search for non-uniform quantization parameters from second-order information. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'hqq' (a proxy recorded while the enum lacked a legal value) and is now 'squeezellm'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)