Stateful Large Language Model Serving with Pensieve

LLM serving systems are stateless across requests, so a growing multi-turn conversation log is re-processed on every turn.

Conversation-state caching: keep the already-processed history across requests in a MULTI-TIER cache that uses both GPU and CPU memory, and generalize the PagedAttention kernel to support attention between multiple input tokens against a GPU cache spread over NON-CONTIGUOUS memory (so a cached prefix need not be a single contiguous block). This is a storage/tiling contribution, NOT quantization or sparsity.

Paper's own claim (abstract): 1.14x-3.0x THE THROUGHPUT OF vLLM AND TensorRT-LLM, and significantly reduced latency, for multi-turn conversation serving. Baseline is explicit and named (vLLM, TensorRT-LLM) which makes this one of the better-anchored claims in the serving literature. COMPRESSION RATIO: none - this is reuse, not compression. QUALITY COST: exactly zero, because the cached history is the same computation.

SLUG RENAMED 2026-10-04 (parent agent): was 'pensieve', colliding with the quantization record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs ([]) all mean the quantization record, so none needed repointing; papers are cited via `sources`, never by bare slug. arXiv ID 2312.05516 verified via /abs/; v3. Venue EuroSys 2024 from Semantic Scholar; year = venue year 2024 (v1 2023-12-09). Recorded in this slice because multi-turn state reuse is a compression method in the serving sense even though it introduces no quantization; do not conflate it with KV-quantization records. arXiv ID and title verified by fetching https://arxiv.org/abs/2312.05516 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2312.05516) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: eurosys has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.