CachePrune: Privacy-Aware and Fine-Grained KV Cache Sharing for Efficient LLM Inference

Argues prompts are mostly privacy-irrelevant segments (system instructions, public material) and that token-level, sensitivity-masked cache sharing removes the direct reuse side channel while cutting TTFT 4.5x and raising hit rate 44% versus the coarse-grained state of the art.

paper

2026-05-22

2026-10-04