SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

Source for chunked prefill as a named mechanism rather than folklore: the workload claim is that a long prompt's single prefill phase stalls the decode phase (inter-token latency spikes), and the fix is to split the prefill into chunks and attach them to otherwise memory-bound decode iterations so the compute-rich prefill tokens raise arithmetic intensity for the whole step. This is the 'one workload, two flop profiles' observation that the multi-turn/prefix-cache and continuous-batching records lean on. | SOURCE AUDIT 2026-10-03: the 'publisher' field held a paragraph of prose (authors + affiliations, 82 chars) instead of a publisher name; trimmed to 'arXiv' with the authorship detail moved here.