inference/research
verifiedmedium · 0.78updated 2026-10-04

InfiniGen (speculative KV prefetch for offloaded serving) - NOT KV quantization

none - no bits reduced. Reduces how much of the KV cache must be FETCHED from host memory.

Fields

id
quantization/infinigen-kv
Scheme
kv-sparsity
Bits
none - no bits reduced. Reduces how much of the KV cache must be FETCHED from host memory.
Activation scheme
None. Performs a 'minimal rehearsal' using the current layer's inputs plus part of the NEXT layer's query weight and key cache to speculate which tokens will matter for the following attention layer, then prefetches only those KV entries.
Kernels
Designed to synergize with modern offloading-based inference systems; improves such a system by up to 3.00x over prior KV cache management methods.
Quality delta
From the abstract: improves overall performance of a modern offloading-based system by up to 3.00x compared to prior KV cache management methods, while offering 'substantially better model accuracy'. No accuracy number given; none recorded.
notes
GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: page/head-level selection of which KV to keep or prefetch. No weights are re-gridded. Boundary record. InfiniGen is often mis-filed under KV quantization because of the speculative-decoding framing, but it neither quantizes nor compresses - it changes WHICH entries cross the PCIe boundary, using the next layer's partial query to predict them. That is a fetch-scheduling result, and it composes with quantization rather than competing: quantize the cache with KIVI and use InfiniGen-style prediction to avoid fetching all of what remains. It is also a reminder that at long context the bottleneck is frequently KV TRANSFER, not KV CAPACITY, and the two need different fixes. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-sparsity'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: page/head-level selection of which KV to keep or prefetch. No weights are re-gridded. Boundary record. InfiniGen is often mis-filed under KV quantization because of the speculative-decoding framing, but it neither quantizes nor compresses - it changes WHICH entries cross the PCIe boundary, using the next layer's partial query to predict them. That is a fetch-scheduling result, and it composes with quantization rather than competing: quantize the cache with KIVI and use InfiniGen-style prediction to avoid fetching all of what remains. It is also a reminder that at long context the bottleneck is frequently KV TRANSFER, not KV CAPACITY, and the two need different fixes. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-sparsity'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)