inference/research
verifiedhigh · 0.85updated 2026-10-03

GPTQ (one-shot approximate second-order PTQ)

w4a16 headline; also reported at w3a16 (and 2-bit / ternary) in the paper's extreme regime. Weight-only - activations are NOT quantized.

Fields

id
quantization/gptq
Scheme
gptq
Bits
w4a16 headline; also reported at w3a16 (and 2-bit / ternary) in the paper's extreme regime. Weight-only - activations are NOT quantized.
Weight group size
128
Kernels
vLLM gptq_marlinvLLM exllama (exllamav2 kernels)MacheteGPTQModel GEMMTensorRT-LLM GPTQ plugins
Quality delta
From the GPTQ paper: OPT-175B and BLOOM-176B quantized to 3-4 bits in ~4 GPU hours with 'minimal increase in perplexity'; the bespoke execution kernels give ~3.25x on A100 and 4.5x on A6000 (quant-gptq-paper). The paper's Figure 1 is a ppl-vs-RTN comparison against the FP16 baseline and is a figure, not tabulated numbers, so no single delta value is recorded here. ADDITIONAL SOURCED FIGURES (qlab-gptq-paper, arXiv 2210.17323 read directly): the paper quantizes 175B-parameter models in approximately four GPU hours, reducing bitwidth to 3 or 4 bits per weight with negligible accuracy degradation, and 'allows us for the first time to execute a 175 billion-parameter model inside a single GPU'. It also explicitly claims reasonable accuracy in the extreme regime 'in which weights are quantized to 2-bit or even ternary quantization levels' - this is the primary source for the GPTQ 3-bit and ternary results, and the reason GPTQ belongs in the sub-4-bit family even though its headline is 4-bit. From the reference implementation (qlab-gptq-github), LLaMa-7B Wiki2 perplexity: FP16 5.68, 4bit-GPTQ 6.09, 3bit-GPTQ 8.07, 3bit group-128 GPTQ 6.61. The last pair is the load-bearing comparison: 3-bit works or does not depending on GROUP SIZE, not on the algorithm. Both are authors' own numbers, not independent replication.
notes
Weight-only, 16-bit compute. GPTQ's contribution is the rounding criterion (approximate inverse Hessian from calibration), not a new number format - a GPTQ file is INT4/INT3 with per-group scales, the same layout AWQ produces, which is why the two share every serving kernel. Group size 128 is the conventional default; 32/64/1024 are legal and change both quality and kernel eligibility, which is a frequent source of 'GPTQ is slow' reports. EXL2 is a GPTQ derivative with per-tensor variable bitrate and should be read alongside this record. exllamav2's own throughput table (3090 Ti / 4090) lists GPTQ at grpsz 128 at 181 and 205 tok/s for Llama-7B (quant-exl2-readme) - a third-party number for the runtime, not a quality delta. UPGRADE (2026-10-03): the bits field previously said only 'w4a16', which understated the method's range - the primary paper claims 3-bit, 2-bit and even TERNARY results, so GPTQ is not merely a 4-bit method. Also added: the reference implementation quantizes columns in order of decreasing activation size (--act-order), which is an activation-AWARE weight-ordering trick and the reason GPTQ can be called 'activation-aware' without quantizing activations. Confirmed from the repo README that the shipped CUDA kernel is a 3-bit MATRIX-VECTOR product kernel, which is the same low-batch caveat as the EXL2 note already here. The parent asked to record GPTQ's 3-bit results: they are now in quality_delta with the exact perplexity figures and the explicit caveat that group-128 is what makes 3-bit usable. Compare the aqlm and squeezellm records for the other way to reach sub-4-bit - changing the representation rather than the grid. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

Weight-only, 16-bit compute. GPTQ's contribution is the rounding criterion (approximate inverse Hessian from calibration), not a new number format - a GPTQ file is INT4/INT3 with per-group scales, the same layout AWQ produces, which is why the two share every serving kernel. Group size 128 is the conventional default; 32/64/1024 are legal and change both quality and kernel eligibility, which is a frequent source of 'GPTQ is slow' reports. EXL2 is a GPTQ derivative with per-tensor variable bitrate and should be read alongside this record. exllamav2's own throughput table (3090 Ti / 4090) lists GPTQ at grpsz 128 at 181 and 205 tok/s for Llama-7B (quant-exl2-readme) - a third-party number for the runtime, not a quality delta. UPGRADE (2026-10-03): the bits field previously said only 'w4a16', which understated the method's range - the primary paper claims 3-bit, 2-bit and even TERNARY results, so GPTQ is not merely a 4-bit method. Also added: the reference implementation quantizes columns in order of decreasing activation size (--act-order), which is an activation-AWARE weight-ordering trick and the reason GPTQ can be called 'activation-aware' without quantizing activations. Confirmed from the repo README that the shipped CUDA kernel is a 3-bit MATRIX-VECTOR product kernel, which is the same low-batch caveat as the EXL2 note already here. The parent asked to record GPTQ's 3-bit results: they are now in quality_delta with the exact perplexity figures and the explicit caveat that group-128 is what makes 3-bit usable. Compare the aqlm and squeezellm records for the other way to reach sub-4-bit - changing the representation rather than the grid. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)