Unlocking Longer Generation with Key-Value Cache Quantization (Hugging Face blog, 2024-05-16)

Author Raushan Turganbay, Hugging Face. Documents the Transformers quantized KV cache (quanto and HQQ backends), inspired by KIVI but with both keys and values quantized per-token plus a residual full-precision cache. LongBench table for Llama-2-7b-chat: Transformers fp16 vs Quanto int4 (nbits=4, group_size=64, residual_length=128, per_token=True): TREC 63.0 vs 63.0, SAMSum 41.12 vs 41.3, TriviaQA 84.28 vs 84.76, HotPotQA 30.08 vs 30.04, Passage_retrieval_en 8.5 vs 9.5. Quanto int2 drops hard (SAMSum 14.04, TriviaQA 63.64). The post states int4 offers roughly a 2.5x memory saving and that, with Flash Attention, a quantized KV cache fits 128k tokens on an 80GB A100 where fp16 fits 40k. Perplexity numbers are a plot only, not a table, so they are not quoted. The markdown source was also read from github.com/huggingface/blog on 2026-10-18.