inference/research
verifiedmedium · 0.75updated 2026-10-04

QServe SmoothAttention (4-bit KV cache with an accuracy-preserving correction)

4-bit KV cache (keys and values), the 'KV4' of W4A8KV4

Fields

id
quantization/qserve-kv4
Scheme
kv-cache-quant
Bits
4-bit KV cache (keys and values), the 'KV4' of W4A8KV4
Activation scheme
Not an activation scheme - 4-bit KV cache mitigated by SmoothAttention, which the paper introduces specifically to counter the accuracy degradation that 4-bit KV quantization causes. The system then makes fused attention memory-bound specifically to capture the KV4 bandwidth win.
Kernels
QServe inference library; fused attention kernels made memory-bound to exploit the KV4 reduction.
Quality delta
Throughput figures only in the abstract (LLaMA-3-8B 1.2x on A100 / 1.4x on L40S vs TensorRT-LLM; Qwen1.5-72B 2.4x on A100 / 3.5x on L40S). No KV-specific quality number in the abstract.
notes
GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: this scheme quantizes the KV cache, not the weights, so there are no weight groups to size. A number in this field would be a category error. (Where the underlying method has a real grouping - e.g. KIVI per-channel keys, KVQuant per-channel - that lives in the method's own record, not in weight_group_size.) Third distinct KV record tied to a weight-quantization paper, and the reason all three are kept separate: QServe's KV contribution is a CORRECTION METHOD (SmoothAttention) rather than a new grouping scheme. That is a genuinely different lever from KIVI's per-channel/per-token axis choice, and it is the one that matters at scale-serving batch sizes where KIVI's low-memory benefit has to coexist with high arithmetic intensity. Note the whole family (KIVI 2/4, KVQuant 3/3, QuaRot 4/4, QServe 4/4, ZipCache mixed) disagrees about the right key bit width - there is no settled answer, and the repo should not imply one by presenting a single record as canonical. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'wq4a4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-cache-quant'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: this scheme quantizes the KV cache, not the weights, so there are no weight groups to size. A number in this field would be a category error. (Where the underlying method has a real grouping - e.g. KIVI per-channel keys, KVQuant per-channel - that lives in the method's own record, not in weight_group_size.) Third distinct KV record tied to a weight-quantization paper, and the reason all three are kept separate: QServe's KV contribution is a CORRECTION METHOD (SmoothAttention) rather than a new grouping scheme. That is a genuinely different lever from KIVI's per-channel/per-token axis choice, and it is the one that matters at scale-serving batch sizes where KIVI's low-memory benefit has to coexist with high arithmetic intensity. Note the whole family (KIVI 2/4, KVQuant 3/3, QuaRot 4/4, QServe 4/4, ZipCache mixed) disagrees about the right key bit width - there is no settled answer, and the repo should not imply one by presenting a single record as canonical. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'wq4a4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-cache-quant'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)