inference/research
verifiedmedium · 0.70updated 2026-10-04

LLM-QAT KV cache quantization (4-bit KV inside a QAT regime)

4-bit KV cache, alongside quantized weights and activations

Fields

id
quantization/llm-qat-kv-cache
Scheme
kv-cache-quant
Bits
4-bit KV cache, alongside quantized weights and activations
Activation scheme
Not an activation scheme - the KV cache is quantized as part of quantization-aware training with straight-through estimators, so the network is trained knowing the cache will be 4-bit. The paper treats KV quantization as essential rather than optional, on the grounds that it is what actually enables long-sequence throughput at current model sizes.
Kernels
Research only; no shipping kernel.
notes
GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: this scheme quantizes the KV cache, not the weights, so there are no weight groups to size. A number in this field would be a category error. (Where the underlying method has a real grouping - e.g. KIVI per-channel keys, KVQuant per-channel - that lives in the method's own record, not in weight_group_size.) Separate record for the KV dimension of LLM-QAT because KV-cache quantization is a distinct decision from weight quantization and the repo's schema has no way to express 'quantizes the KV cache' except through bits and activation_scheme free text. Do not fold this into the llm-qat record: that record is about the training regime, this one is about the cache format, and a reader deciding how many bits to spend on KV should not have to read a QAT paper to find out. Symmetric key/value bits here (4 and 4) - the asymmetric alternative is KIVI's 2-bit-key / 4-bit-value. Request the parent add a kv-cache enum family so records like this one stop needing a weight-format enum value as a stand-in. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-cache-quant'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: this scheme quantizes the KV cache, not the weights, so there are no weight groups to size. A number in this field would be a category error. (Where the underlying method has a real grouping - e.g. KIVI per-channel keys, KVQuant per-channel - that lives in the method's own record, not in weight_group_size.) Separate record for the KV dimension of LLM-QAT because KV-cache quantization is a distinct decision from weight quantization and the repo's schema has no way to express 'quantizes the KV cache' except through bits and activation_scheme free text. Do not fold this into the llm-qat record: that record is about the training regime, this one is about the cache format, and a reader deciding how many bits to spend on KV should not have to read a QAT paper to find out. Symmetric key/value bits here (4 and 4) - the asymmetric alternative is KIVI's 2-bit-key / 4-bit-value. Request the parent add a kv-cache enum family so records like this one stop needing a weight-format enum value as a stand-in. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-cache-quant'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)