inference/research
verifiedmedium · 0.80updated 2026-10-18

ExLlamaV2 Q4/FP8 KV cache evaluation (qcache_eval.md, Hadamard Q4)

repo

Fields

id
sources/exllamav2-qcache-eval-q4-fp8
Publisher
turboderp-org / ExLlamaV2
Kind
repo
Published
2024-03
Accessed
2026-10-18
notes
Fetched the raw doc 2026-10-18 from https://raw.githubusercontent.com/turboderp-org/exllamav2/master/doc/qcache_eval.md. Author evaluation, not a paper. Perplexity is token-level on The Pile, 10 rows of 512 tokens. The current (post-0.0.18) method applies Hadamard rotations to keys and values. Llama2 7B FP16 row: FP16 cache 10.91, FP8 cache 11.24, Q4 cache 11.05 (old Q4 was 11.16). Mistral 7B Instruct FP16 row: FP16 cache 12.69, FP8 12.71, Q4 12.69. HumanEval pass@1 was measured on the pre-0.0.18 Q4 and is not used as the quality number. No context length and no tok/s are stated for the cache modes.

Notes

Fetched the raw doc 2026-10-18 from https://raw.githubusercontent.com/turboderp-org/exllamav2/master/doc/qcache_eval.md. Author evaluation, not a paper. Perplexity is token-level on The Pile, 10 rows of 512 tokens. The current (post-0.0.18) method applies Hadamard rotations to keys and values. Llama2 7B FP16 row: FP16 cache 10.91, FP8 cache 11.24, Q4 cache 11.05 (old Q4 was 11.16). Mistral 7B Instruct FP16 row: FP16 cache 12.69, FP8 12.71, Q4 12.69. HumanEval pass@1 was measured on the pre-0.0.18 Q4 and is not used as the quality number. No context length and no tok/s are stated for the cache modes.

Cross-references

This record references no other records.