KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (arXiv:2401.18079)

Fields

id

sources/mkt-kvquant-neurips2024

URL

https://arxiv.org/abs/2401.18079

Publisher

arXiv

Kind

paper

Published

2024-01-31

Accessed

2026-10-04

notes

NeurIPS 2024 (per arXiv comments field). KV cache quantization with per-channel key quantization, pre-RoPE key quantization, non-uniform KV cache quantization, and per-vector dense-and-sparse quantization. Achieves <0.1 perplexity degradation with 3-bit quantization. Enables LLaMA-7B with 1M context on single A100-80GB and 10M on 8-GPU system. Up to ~1.7x speedups for LLaMA-7B. Custom CUDA kernels.

Notes

NeurIPS 2024 (per arXiv comments field). KV cache quantization with per-channel key quantization, pre-RoPE key quantization, non-uniform KV cache quantization, and per-vector dense-and-sparse quantization. Achieves <0.1 perplexity degradation with 3-bit quantization. Enables LLaMA-7B with 1M context on single A100-80GB and 10M on 8-GPU system. Up to ~1.7x speedups for LLaMA-7B. Custom CUDA kernels.

Cross-references

This record references no other records.