inference/research
verifiedmedium · 0.80updated 2026-10-04

QServe / QoQ (W4A8KV4 quantization and system co-design)

w4a8kv4 (4-bit weights, 8-bit activations, 4-bit KV cache) - explicitly named QoQ, quattuor-octo-quattuor, i.e. 4-8-4

Fields

id
quantization/qserve-w4a8kv4
Scheme
w4a8kv4
Bits
w4a8kv4 (4-bit weights, 8-bit activations, 4-bit KV cache) - explicitly named QoQ, quattuor-octo-quattuor, i.e. 4-8-4
Activation scheme
8-bit activations, kept cheap by PROGRESSIVE QUANTIZATION, which allows low dequantization overhead in the W4A8 GEMM. The KV cache is quantized to 4 bits with SmoothAttention, a technique added specifically to mitigate the accuracy degradation 4-bit KV quantization causes.
Kernels
QServe inference library (QoQ). Compute-aware weight reordering, register-level parallelism to cut dequantization latency, and fused attention made memory-bound to exploit the KV4 gain.
Quality delta
Measured quality from Table 2 and Table 3 of the paper (arXiv:2405.04532, qlab-qserve-paper): WikiText-2 perplexity (seqlen 2048) on Llama-3-8B is 6.70 (W4A8KV4 g128) and 6.81 (W4A8KV4 per-channel) vs 6.14 FP16 baseline (+0.56 to +0.67 ppl delta); on Llama-2-7B is 5.67 (g128) and 5.75 vs 5.47 FP16 (+0.20 to +0.28 ppl delta); on Llama-2-70B is 3.46 (g128) vs 3.32 FP16 (+0.14 ppl delta). Downstream 5-task zero-shot average (PIQA, ARC-e, ARC-c, HellaSwag, WinoGrande) on Llama-2-7B is 67.95% (g128) and 67.57% vs 68.98% FP16 (-1.03 to -1.41 pp delta). On LongBench (Llama-3.1-8B-Instruct, Table 5), QoQ W4A8KV4 g128 scores 38.38 average vs 38.52 BF16 (-0.14 delta).
notes
GROUP SIZE LEFT NULL 2026-10-04: the QServe/QoQ record describes the W4A8KV4 SCHEME and its system co-design (SmoothAttention, token-wise smoothing), not one concrete weight grid. The KV group detail lives in the sibling qserve-kv4 record. Enum mismatch, and this one matters: recorded as 'wq4a4' because that is the nearest 4-bit-weight value, but QServe is W4A8 - activations are 8-bit, not 4-bit. Anyone reading this as a W4A4 method would be wrong; bits field says w4a8kv4 and this note says so explicitly. Request the parent add 'w4a8kv4'. The real contribution is SYSTEM co-design: it identifies that INT4 serving on GPUs is bottlenecked not by the GEMM but by dequantization on low-throughput CUDA cores, which is the same class of problem as the w4a16-vs-w4a4 distinction - INT4 weight-only buys memory, not compute, and unless you also fix the dequant path it can be slower than fp16. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'wq4a4' (a proxy recorded while the enum lacked a legal value) and is now 'w4a8kv4'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: W4A4/W8A8 RESEARCH FORMAT - this empty list is a REAL FINDING about the absence of a silicon path, not a tautology. W4A8 - 8-bit activations, not 4-bit; no 4-bit activation silicon path exists for it to use. Consequence for a reader: this is the class where a vendor could in principle have shipped something later, so it is the only class worth re-checking against a new hardware release. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE LEFT NULL 2026-10-04: the QServe/QoQ record describes the W4A8KV4 SCHEME and its system co-design (SmoothAttention, token-wise smoothing), not one concrete weight grid. The KV group detail lives in the sibling qserve-kv4 record. Enum mismatch, and this one matters: recorded as 'wq4a4' because that is the nearest 4-bit-weight value, but QServe is W4A8 - activations are 8-bit, not 4-bit. Anyone reading this as a W4A4 method would be wrong; bits field says w4a8kv4 and this note says so explicitly. Request the parent add 'w4a8kv4'. The real contribution is SYSTEM co-design: it identifies that INT4 serving on GPUs is bottlenecked not by the GEMM but by dequantization on low-throughput CUDA cores, which is the same class of problem as the w4a16-vs-w4a4 distinction - INT4 weight-only buys memory, not compute, and unless you also fix the dequant path it can be slower than fp16. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'wq4a4' (a proxy recorded while the enum lacked a legal value) and is now 'w4a8kv4'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: W4A4/W8A8 RESEARCH FORMAT - this empty list is a REAL FINDING about the absence of a silicon path, not a tautology. W4A8 - 8-bit activations, not 4-bit; no 4-bit activation silicon path exists for it to use. Consequence for a reader: this is the class where a vendor could in principle have shipped something later, so it is the only class worth re-checking against a new hardware release. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)