inference/research
verifiedmedium · 0.75updated 2026-10-04

ZeroQuant v1 (fine-grained hardware-friendly PTQ + layer-by-layer distillation)

w8a8 headline; also int4 weights in fully-connected modules with int8 weights in attention and int8 activations (a mixed w4a8 configuration)

Fields

id
quantization/zeroquant-v1
Scheme
zeroquant
Bits
w8a8 headline; also int4 weights in fully-connected modules with int8 weights in attention and int8 activations (a mixed w4a8 configuration)
Activation scheme
Fine-grained hardware-friendly per-token/per-channel dynamic quantization for activations, plus layer-by-layer knowledge distillation (LKD) that works without access to the original training data - LKD is regenerated from the model itself, so 'data-free' here means no corpus, not no compute.
Kernels
Purpose-built quantized CUDA backend in the paper, described as removing quantization/dequantization overhead.
Quality delta
From the paper's abstract: INT8 weight+activation is cost-free for BERT and GPT-3 style models with minimal accuracy impact, giving up to 5.19x (BERT) and 4.16x (GPT-3 style) speedup over FP16; on GPT-J 6B and GPT-NeoX 20B the INT8 model achieves accuracy similar to FP16 with up to 5.2x better efficiency. These are speed figures, not a perplexity delta; no PPL number is given in the abstract.
notes
GROUP SIZE LEFT NULL 2026-10-04: ZeroQuant v1's contribution is FINE-GRAINED hardware-friendly quantization - per-token/per-channel granularity chosen for kernel efficiency - plus block-wise input quantization. Its granularity axes are per-token and per-channel, which are not the contiguous weight-group axis this field measures. Enum mismatch: 'gptq' is the closest enum - ZeroQuant v1 is weight-only-oriented PTQ in that lineage, and its int4 path is grouped/quantized weights with dequantization back to fp16. Request the parent add 'zeroquant'. The speedups above are from the authors' own system with their own kernels and no independent replication; treat as vendor_claim-style numbers rather than measured third-party results. ZeroQuant v1 predates W4A4 hardware entirely - on any of the accelerators listed under emulated_support, the int8 path is the only one with a plausible tensor-core story, and even that is per-tensor rather than block-scaled. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'gptq' (a proxy recorded while the enum lacked a legal value) and is now 'zeroquant'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE LEFT NULL 2026-10-04: ZeroQuant v1's contribution is FINE-GRAINED hardware-friendly quantization - per-token/per-channel granularity chosen for kernel efficiency - plus block-wise input quantization. Its granularity axes are per-token and per-channel, which are not the contiguous weight-group axis this field measures. Enum mismatch: 'gptq' is the closest enum - ZeroQuant v1 is weight-only-oriented PTQ in that lineage, and its int4 path is grouped/quantized weights with dequantization back to fp16. Request the parent add 'zeroquant'. The speedups above are from the authors' own system with their own kernels and no independent replication; treat as vendor_claim-style numbers rather than measured third-party results. ZeroQuant v1 predates W4A4 hardware entirely - on any of the accelerators listed under emulated_support, the int8 path is the only one with a plausible tensor-core story, and even that is per-tensor rather than block-scaled. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'gptq' (a proxy recorded while the enum lacked a legal value) and is now 'zeroquant'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (6)