TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text

Conventional RAG concatenates many retrieved chunks and prefills all of them online, and since prefill is the most compute-dense step in the whole serving stack, this dominates time-to-first-token.

Inverts the paradigm: precompute and store each document's KV cache OFFLINE, then at inference time directly RETRIEVE the saved KV cache for prefill instead of running the prefill over the chunk text. Online KV computation is eliminated entirely. The non-obvious part is that this is not a no-op, and the paper's actual contribution is the bookkeeping: attention mask semantics (each chunk needs INDEPENDENT attention rather than joint attention over the concatenation) and POSITION ID remapping for the concatenated KV caches, because positions baked into an offline cache are contiguous-per-document and must be remapped to their real slots at query time. The authors then FINE-TUNE the LM to tolerate the resulting attention and position-ID pattern, which is the honest cost of the approach - it is not a training-free swap.

Paper's own claim: reduces time-to-first-token by up to 9.4x versus conventional RAG systems (8.6x on average) across a suite of RAG benchmarks, while reserving comparable accuracy. Two honest qualifiers the reader needs: the baseline is a CONVENTIONAL RAG SYSTEM, not a modern engine with prefix caching enabled, and the comparable-accuracy claim requires the authors' fine-tuning - which is why the paper's own summary lists it as contribution 5 rather than an aside.

RECORDED AS arxiv-preprint WITH venue_track 'unknown' DESPITE BEING WIDELY CITED - and the reason is worth stating because it is the case the task's rules exist for. VENUE SEARCH RESULT: the OpenReview index holds exactly two records for this title: content.venue 'ICLR 2025 Conference Withdrawn Submission' (venueid ICLR.cc/2025/Conference/Withdrawn_Submission) and content.venue 'ACL ARR 2025 May Submission' (venueid aclweb.org/ACL/ARR/2025/May/Submission). Neither is an acceptance. An ARR submission is a REVIEW BATCH, not a publication: a paper can sit in an ARR batch indefinitely without appearing in any proceedings volume, so citing ACL ARR as its venue would be exactly the confident-wrong-answer failure this repo has shipped twice. Corpus checks, all negative: absent from ACL Anthology, absent from PMLR v267 (ICML 2025), absent from MLSys 2025 and 2026, absent from NeurIPS 2024. arXiv comment is EMPTY. So there is NO proceedings entry and NO venue claim - arxiv-preprint/unknown is the honest record, and confidence is capped at 0.8 rather than lowered further only because the arXiv metadata itself is solid. CONTRAST WITH ITS SIBLING: Cache-Craft (same problem class, same year, also by a group that ran real production workloads) DID land in Proc. ACM Manag. Data 2025. Two RAG cache papers of the same vintage, one peer-reviewed and one not, is worth keeping side by side. VENUE CORRECTED 2026-10-05 (wave 6, agent w6-papers): arxiv-preprint/unknown -> EMNLP 2025 MAIN, year 2024 -> 2025, confidence 0.8 -> 0.88. This is a PAPER THAT GOT PUBLISHED SINCE WAVE 4, and wave 4's negative finding was executed correctly and was simply true at the time. WHAT WAVE 4 FOUND, restated because it was right: the OpenReview index held exactly two records for this title, 'ICLR 2025 Conference Withdrawn Submission' and 'ACL ARR 2025 May Submission'; neither is an acceptance; an ARR batch is a REVIEW BATCH, not a publication; and TurboRAG was absent from the ACL Anthology, PMLR v267, MLSys 2025/2026 and NeurIPS 2024. WHY THAT WAS STILL NOT THE LAST WORD - and this is the transferable lesson. The ACL ARR 2025 May submission was not a dead end; it was a REAL submission that was later ACCEPTED. A negative ARR search is not a negative venue finding, it is a statement about the state of a review pipeline on the day you ran it. Recording 'absent from the ACL Anthology' as evidence of non-publication is only valid until the Anthology catches up, and EMNLP 2025 (Suzhou, November 2025) postdates wave 4's sweep. Note the contrast the record already drew with Cache-Craft landing in Proc. ACM Manag. Data 2025: two RAG cache papers of the same vintage, and the outcome diverged - so the honest position was never 'unpublished', it was 'not yet published as of the search date'. THE PUBLISHER'S OWN DEPOSIT (source w6p-aclanthology-2025-emnlp-main-334-turborag): ACL Anthology 2025.emnlp-main.334, volume 'Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing', pages 6588-6601, Suzhou China, November 2025, venue EMNLP, SIG SIGDAT, publisher ACL, DOI 10.18653/v1/2025.emnlp-main.334, bibkey lu-etal-2025-turborag. Identification is EXACT on both axes: the title matches character for character and all FIVE authors (Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, Yaohua Tang) match the arXiv API response for 2410.07590 in the same order. THE CAMERA-READY ABSTRACT DIFFERS FROM THE ARXIV ABSTRACT, and the difference is a MECHANISM claim rather than wording, so it is worth carrying. arXiv v1 says the system 'fine-tune[s] a pretrained language model to maintain model accuracy'; the Anthology version describes a hybrid offline-online paradigm that pre-computes chunk-level KV caches, stitches them at inference using INDEPENDENT-ATTENTION and REORDERED-RoPE techniques, and 'preserves answer quality WITHOUT CHANGING THE MODEL ARCHITECTURE'. So the published version is strictly stronger: no model fine-tuning required. A reader holding only the arXiv PDF gets the weaker story and never sees the two named techniques. The 9.4x max / 8.6x mean TTFT reduction against conventional RAG is identical in both.