inference/research
verifiedmedium · 0.80updated 2026-10-04

Atom (mixed-precision block-outlier low-bit quantization for serving)

w4a4 (4-bit weights AND 4-bit activations) with mixed-precision outlier channels retained at higher precision

Fields

id
quantization/atom
Scheme
atom
Bits
w4a4 (4-bit weights AND 4-bit activations) with mixed-precision outlier channels retained at higher precision
Weight group size
128
Activation scheme
4-bit activations with a mixed-precision, fine-grained process: a small set of outlier channels/blocks is kept at higher precision while the dense bulk is 4-bit. The paper's motivation is explicitly that 8-bit weight-activation schemes cannot leverage modern GPUs' 4-bit integer operators, so it targets the 4-bit regime to hit real hardware paths.
Kernels
Purpose-built CUDA kernels with low-bit operators, evaluated on A800-class hardware in the paper. Faster than the INT8 baseline because it actually uses the 4-bit integer operators.
Quality delta
From the abstract: end-to-end throughput up to 7.73x versus FP16 and 2.53x versus INT8 quantization at the same latency target, with negligible accuracy loss, in the serving context. These are throughput figures from the authors' own system on their own kernels - vendor-style numbers, not independent. No perplexity figure in the abstract.
notes
FILLED 2026-10-04 from the paper (arXiv 2310.19102v3), not from convention: 'We evaluate Atom using a group size of 128', 'For example, a group size of 128 implies that every contiguous sequence of 128 elements is treated as a single group, which is quantized independently', and 'With a group size of 128 and a high precision channel size of 128, Atom has an effective bit of 4'. Note the SECOND 128: Atom has BOTH a 128-element quant group AND a 128-channel high-precision outlier group, and the 4-bit effective figure only holds with both fixed together. This is one of the clearest instances of the distinction the repo cares about. Atom is W4A4 and it DOES have a hardware story, because it is built around 4-bit integer operators - but that is an integer-operator path on datacenter GPUs, not the FP4 tensor-core path that Blackwell introduced. Contrast nvfp4, which is W4A4 with a genuine FP4 tensor-core path. Atom's outlier channels at higher precision are the mixed-precision escape hatch that keeps 4-bit activations workable without needing to learn a rotation first; compare LLM.int8() and QuaRot, which solve the same problem differently. Enum matches ('wq4a4') because it is literally W4A4; no mismatch to report here. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'wq4a4' (a proxy recorded while the enum lacked a legal value) and is now 'atom'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: W4A4/W8A8 RESEARCH FORMAT - this empty list is a REAL FINDING about the absence of a silicon path, not a tautology. its own record states it is faster than the INT8 baseline BECAUSE it uses 4-bit integer operators - an empirical speedup, not an architectural path. Consequence for a reader: this is the class where a vendor could in principle have shipped something later, so it is the only class worth re-checking against a new hardware release. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

FILLED 2026-10-04 from the paper (arXiv 2310.19102v3), not from convention: 'We evaluate Atom using a group size of 128', 'For example, a group size of 128 implies that every contiguous sequence of 128 elements is treated as a single group, which is quantized independently', and 'With a group size of 128 and a high precision channel size of 128, Atom has an effective bit of 4'. Note the SECOND 128: Atom has BOTH a 128-element quant group AND a 128-channel high-precision outlier group, and the 4-bit effective figure only holds with both fixed together. This is one of the clearest instances of the distinction the repo cares about. Atom is W4A4 and it DOES have a hardware story, because it is built around 4-bit integer operators - but that is an integer-operator path on datacenter GPUs, not the FP4 tensor-core path that Blackwell introduced. Contrast nvfp4, which is W4A4 with a genuine FP4 tensor-core path. Atom's outlier channels at higher precision are the mixed-precision escape hatch that keeps 4-bit activations workable without needing to learn a rotation first; compare LLM.int8() and QuaRot, which solve the same problem differently. Enum matches ('wq4a4') because it is literally W4A4; no mismatch to report here. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'wq4a4' (a proxy recorded while the enum lacked a legal value) and is now 'atom'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: W4A4/W8A8 RESEARCH FORMAT - this empty list is a REAL FINDING about the absence of a silicon path, not a tautology. its own record states it is faster than the INT8 baseline BECAUSE it uses 4-bit integer operators - an empirical speedup, not an architectural path. Consequence for a reader: this is the class where a vendor could in principle have shipped something later, so it is the only class worth re-checking against a new hardware release. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)