inference/research
verifiedmedium · 0.80updated 2026-10-04

LLM-QAT (data-free quantization-aware training, NVIDIA)

w4a4 to w8a8 plus 4-bit KV cache (per-tensor/per-channel groups configured per experiment)

Fields

id
quantization/llm-qat
Scheme
qat
Bits
w4a4 to w8a8 plus 4-bit KV cache (per-tensor/per-channel groups configured per experiment)
Activation scheme
Quantization-aware training: straight-through estimators over weights AND activations AND the KV cache, with the distillation target being the pretrained model's own greedy/sampled generations rather than any training corpus. This is what makes it 'data-free' and usable on models whose training data is unavailable.
Kernels
Research-only. No shipping kernel named in the paper; evaluated with simulation/fake-quant in the paper's harness.
Quality delta
Paper states 'large improvements over training-free methods, especially in the low-bit settings' for LLaMA 7B/13B/30B down to 4-bit. The abstract gives no numeric perplexity delta and the experiments section was not read this session, so no number is recorded here. Third-party comparisons elsewhere cite LLM-QAT trailing W4A4 by ~22 points of zero-shot accuracy, but that figure is from SpinQuant's table, not from this paper.
notes
GROUP SIZE LEFT NULL 2026-10-04: LLM-QAT is a quantization-AWARE TRAINING recipe spanning w4a4 to w8a8 plus 4-bit KV, so the group size is a free choice per configuration. Its KV-cache variant is the record that carries the KV detail. Enum mismatch, deliberate: the closest existing value is 'ternary' but LLM-QAT is a training regime, not a ternary format - it is reported at 4-bit and above. Request the parent add a 'qat' enum value. The important thing this record captures is the W4A4-vs-W4A16 distinction: LLM-QAT is a genuine W4A4 method (it quantizes activations, unlike GPTQ/AWQ), but its W4A4 has no hardware int4xint4 path - same caveat as QuaRot and SpinQuant. Also note the KV cache is quantized here, which is unusual for a weight-quantization paper and links Task A to Task D. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'ternary' (a proxy recorded while the enum lacked a legal value) and is now 'qat'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: W4A4/W8A8 RESEARCH FORMAT - this empty list is a REAL FINDING about the absence of a silicon path, not a tautology. a quantization-AWARE TRAINING recipe, evaluated with fake-quant/simulation in the paper's own harness. Consequence for a reader: this is the class where a vendor could in principle have shipped something later, so it is the only class worth re-checking against a new hardware release. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE LEFT NULL 2026-10-04: LLM-QAT is a quantization-AWARE TRAINING recipe spanning w4a4 to w8a8 plus 4-bit KV, so the group size is a free choice per configuration. Its KV-cache variant is the record that carries the KV detail. Enum mismatch, deliberate: the closest existing value is 'ternary' but LLM-QAT is a training regime, not a ternary format - it is reported at 4-bit and above. Request the parent add a 'qat' enum value. The important thing this record captures is the W4A4-vs-W4A16 distinction: LLM-QAT is a genuine W4A4 method (it quantizes activations, unlike GPTQ/AWQ), but its W4A4 has no hardware int4xint4 path - same caveat as QuaRot and SpinQuant. Also note the KV cache is quantized here, which is unusual for a weight-quantization paper and links Task A to Task D. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'ternary' (a proxy recorded while the enum lacked a legal value) and is now 'qat'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: W4A4/W8A8 RESEARCH FORMAT - this empty list is a REAL FINDING about the absence of a silicon path, not a tautology. a quantization-AWARE TRAINING recipe, evaluated with fake-quant/simulation in the paper's own harness. Consequence for a reader: this is the class where a vendor could in principle have shipped something later, so it is the only class worth re-checking against a new hardware release. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (6)