Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv 2403.02310)

arXiv v3 dated 2024-06-17, cats cs.LG/cs.DC. The arXiv comment field is EMPTY (no venue claim). Venue verified independently from the USENIX OSDI '24 technical-sessions page, which lists 'Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve' in its program. Chunked prefill is the paper's central mechanism.