id
kv-efficiency/llamacpp-gguf-q8kv-llama31-8b
Model
Llama-3.1-8B-Instruct (GGUF)
KV bytes basis
32 layers x 8 kv heads x 128 head_dim x 2 (K+V) x 1 byte (q8_0 stores int8 + a per-block fp16 scale; the scale adds ~2% which the byte/token field rounds past) = 65,536 B/token. The fp16 model record carries 131,072 B/token — q8_0 KV halves it, and at 32K context the KV pool drops ~4 GiB to ~2 GiB per sequence.
Prefix reuse
context-checkpoint
Methodology
The KV-quant arm of the unified llama.cpp quantization study (arXiv:2601.14277, 'Which Quantization Should I Use?', Kurt 2026): WikiText-2 ppl and a 5-benchmark suite on Llama-3.1-8B-Instruct, quantized KV read at -ctk q8_0 -ctv q8_0, the llama.cpp default KV-quant tier. Reported quality cost ~1% — the source's own number, which matches the research file's '~1% for Q8_0 KV' row. context_length 32768 chosen as the repo's mid-range comparison point; the study's own protocol is fixed-length ppl + benchmark scoring rather than a TTFT/tok-s sweep, so measured_* stay null rather than borrowing the CPU pp512/tg128 figures the study reports for WEIGHT quants (different quantity).
notes
Recorded because q8_0 KV is the single most-deployed KV-quant in this domain — it is a llama.cpp flag, not a paper — and because its 2x compression at ~1% quality is the safe default the research file recommends for the 8B-on-consumer-GPU class. prefix_reuse context-checkpoint names llama.cpp's KV snapshot mechanism (up to 32 named checkpoints per slot), which is how that engine reuses prefixes: weaker than APC/radix (per-slot, not shared across requests) but the honest enum for what the engine actually does. The deeper GGUF KV tiers (q4_0/q4_1/iq4_nl/q5_0/q5_1 at 4-5 bits, 1-3% quality) are covered by the same source; a separate record per tier was deliberately not seeded — the schema supports them but the study reads q8_0 as the tier with an actual measurement behind it.