KV-cache efficiency (39)
SGLang RadixAttention hit on 8x RTX PRO 6000: 112K-token shared prefix served in 0.34 s (~39x the cold prefill rate)
kv-efficiency/sglang-radix-hit-112k-prefix
draftmedium · 0.75
SGLang serving DeepSeek-V4.1-Flash on 8x RTX PRO 6000: fp8 KV cache, 128K cold prefill at chunked_prefill_size=2048
kv-efficiency/sglang-dsv41-fp8kv-128k-cold-prefill
draftmedium · 0.80
TensorRT-LLM FP4/NVFP4 KV cache on Blackwell: the 4-bit KV tier with a hardware path, <2% reported quality hit
kv-efficiency/trtllm-nvfp4-kv-blackwell
draftmedium · 0.70
DeepSeek-V3 MLA at 128K: 68.6 KiB/token latent KV, ~85x smaller than the MHA-equivalent — the reason 128K fits at all
kv-efficiency/deepseek-v3-mla-vs-mha-128k
verifiedhigh · 0.90
ExLlamaV2 Hadamard Q4 KV on Llama-2-7B FP16: 4x smaller cache, +0.14 Pile perplexity
kv-efficiency/exllamav2-q4-kv-llama2-7b-fp16
verifiedmedium · 0.78
Gemma 2 9B alternating sliding window at 128K: 177,408 B/token effective, 1.94x under the naive full cache
kv-efficiency/gemma2-9b-alternating-swa-128k
verifiedmedium · 0.84
Gemma 3 4B sliding-window hybrid at 32K: Google measures 4.7 GB of 8-bit KV, 4.8x under a full-context cache
kv-efficiency/gemma3-4b-swa-32k-measured-kv
verifiedmedium · 0.82
Gemma 3n E2B shared-KV sliding window at 32K
kv-efficiency/gemma3n-e2b-shared-kv-swa-32k
verifiedmedium · 0.74
Gemma 4 26B-A4B on one RTX 3090 at 256K: 83.0 tok/s at 128K and 64.4 tok/s at 256K, the sliding window holding KV to 21,280 B/token
kv-efficiency/llamacpp-gemma4-26b-a4b-rtx3090-256k
verifiedmedium · 0.80
Gemma 4 26B-A4B sliding-window hybrid at 128K: 22,080 B/token effective, 11.1x under a full-context GQA cache
kv-efficiency/gemma4-26b-a4b-swa-128k-effective-kv
verifiedhigh · 0.90
Granite 4.0 H Tiny hybrid at 128K: 4 attention layers of 40, 8,192 B/token, 10x under full GQA
kv-efficiency/granite-4-h-tiny-hybrid-128k-kv
verifiedhigh · 0.86
KIVI asymmetric 2-bit KV (2-bit keys / 4-bit values) on Llama-2-class GQA models: ~8x KV compression for a ~2-5% perplexity delta
kv-efficiency/kivi-2bit-llama2-gqa-4k
verifiedhigh · 0.85
KIVI-2 on Llama-2-13B MHA: 4x smaller KV than bf16, GSM8K 22.67 to 20.77
kv-efficiency/kivi-2bit-llama2-13b-mha-4k
verifiedhigh · 0.85
KIVI-2 on Llama-3-8B-Instruct: 2-bit KV, LongBench average 45.21 to 44.37, 7.1x smaller than bf16 at the 8K window
kv-efficiency/kivi-2bit-llama3-8b-gqa-8k
verifiedhigh · 0.85
KVQuant nuq3-1% on LLaMA-7B: 4.8x KV compression, under 0.1 WikiText-2 perplexity, A6000 decode kernel faster than fp16
kv-efficiency/kvquant-nuq3-llama7b-a6000
verifiedhigh · 0.86
llama.cpp GGUF q8_0 KV cache (-ctk q8_0 -ctv q8_0) on Llama-3.1-8B: halves KV to 64 KiB/token, ~1% quality cost
kv-efficiency/llamacpp-gguf-q8kv-llama31-8b
verifiedhigh · 0.85
llama.cpp q4_0 KV on Qwen3-8B: 3.6x smaller cache, +1% perplexity, half the decode
kv-efficiency/llamacpp-qwen3-8b-q4kv-32k-a100
verifiedmedium · 0.80
LMCache TurboQuant 4-bit KV host-DRAM tier hit rate
kv-efficiency/lmcache-turboquant-4bit-dram-tier-100k
verifiedmedium · 0.65
MiMo-V2.6-Flash sliding-window hybrid at 128K: 23,040 B/token growing, 5.3x under a full-context cache
kv-efficiency/mimo-v26-flash-swa-128k-effective-kv
verifiedhigh · 0.85
Mistral-7B rolling KV buffer at 32K: 16,384 B/token effective, 8x under a full-context GQA cache
kv-efficiency/mistral-7b-swa-32k-rolling-kv
verifiedhigh · 0.90
PaliGemma 3B text tower MQA: 1 KV head of 8, 18,432 B/token, 8x under full MHA
kv-efficiency/paligemma-3b-mqa-8x-kv
verifiedmedium · 0.74
Qwen3-8B Q4_K on an RTX 4060 Ti 16GB at 64K: 13.0 tok/s decode, 392.1 tok/s prefill, f16 KV at 144 KiB/token
kv-efficiency/llamacpp-qwen3-8b-rtx4060ti-64k
verifiedmedium · 0.70
Qwen3-8B Q4_K on an RTX 4070 SUPER 12GB at 32K: 42.2 tok/s decode, and the 64K cell is empty
kv-efficiency/llamacpp-qwen3-8b-rtx4070s-32k
verifiedmedium · 0.70
Qwen3-8B Q4_K on an RTX 4080 16GB at 64K: 39.0 tok/s decode, 937.9 tok/s prefill, f16 KV at 144 KiB/token
kv-efficiency/llamacpp-qwen3-8b-rtx4080-64k
verifiedmedium · 0.70
Qwen3-8B Q4_K on an RTX 5060 Ti 16GB at 64K: 20.7 tok/s decode, 650.3 tok/s prefill, f16 KV at 144 KiB/token
kv-efficiency/llamacpp-qwen3-8b-rtx5060ti-64k
verifiedmedium · 0.70
Qwen3.5-27B hybrid KV on one RTX 3090 at 64K: 16,384 B/token, 28.8 tok/s where a full-GQA cache would not fit
kv-efficiency/llamacpp-qwen35-27b-rtx3090-64k
verifiedmedium · 0.75
Qwen3.5-35B-A3B hybrid (linear + every-4th full attention) with FP8 KV on vLLM: 20.5 KiB/token base, fully recovers 1M-context AUC in the fp8-KV validation
kv-efficiency/vllm-qwen35-35b-hybrid-fp8kv-256k
verifiedhigh · 0.85
Qwen3.5-35B-A3B hybrid KV at 128K versus full GQA
kv-efficiency/qwen35-35b-a3b-hybrid-kv-128k
verifiedhigh · 0.85
Qwen3.5-9B hybrid KV at 40K, measured decode vs dense Qwen3-8B
kv-efficiency/vllm-qwen35-9b-hybrid-kv-40k-measured
verifiedmedium · 0.78
Qwen3.8-27B q4 KV at 197K on an RTX 4090: 26.6 tok/s decode, 121.5 s to first token
kv-efficiency/llamacpp-qwen38-27b-rtx4090-q4kv-197k
verifiedmedium · 0.72
RTX 3090 gpt-oss-20B MXFP4 Flash Attention decode at 131k
kv-efficiency/llama-cpp-gptoss20b-rtx3090-fa-131k
verifiedmedium · 0.65
RTX 3090 Ti, gpt-oss-20B MXFP4 at 128K: 61.5 tok/s decode, hybrid KV at 49,152 B/token
kv-efficiency/llamacpp-gptoss20b-rtx3090ti-128k
verifiedmedium · 0.70
SGLang RadixAttention in Chatbot Arena: 74.1 percent cache hit on Vicuna-33B, 1.7x lower first-token latency
kv-efficiency/sglang-radix-chatbot-arena-vicuna33b-hit
verifiedmedium · 0.80
Transformers Quanto int4 KV on Llama-2-7B-chat: LongBench within 0.5 points of fp16
kv-efficiency/transformers-quanto-int4-kv-llama2-7b-chat
verifiedmedium · 0.66
TurboQuant 3.5-bit KV on Llama-3.1-8B quality-neutral
kv-efficiency/turboquant-llama31-8b-3p5bit-kv
verifiedmedium · 0.70
TurboQuant on 4x RTX 3090: 30 GB of KV freed on Qwen3.5-27B, and 131K context on Qwen3.5-35B-A3B where bf16 OOMs at 32K
kv-efficiency/turboquant-qwen35-35b-a3b-4x3090-3bit
verifiedmedium · 0.68
vLLM automatic prefix caching on one RTX 5090: a 150K Qwen3.8-27B replay falls from 53.2 s to 2.3 s at 93.5 percent hit
kv-efficiency/vllm-qwen38-27b-apc-150k-rtx5090
verifiedlow · 0.62
vLLM FP8 KV (FA3, H100): Llama-3.1-8B decode ITL slope vs context halved vs bf16; 128K MRCR recovers 97-98% AUC
kv-efficiency/vllm-llama31-8b-fp8kv-128k-itl-halved
verifiedhigh · 0.88
vLLM prefix caching on Qwen3-8B: 9.58x at 32K shared, nothing past the cache
kv-efficiency/vllm-qwen3-8b-prefix-reuse-32k-a100
verifiedmedium · 0.76