DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024)

Abstract states the mechanism and the limits: colocating the phases 'leads to strong prefill-decoding interferences' and 'couples the resource allocation and parallelism plans for both phases'; DistServe 'assigns prefill and decoding computation to different GPUs, hence eliminating prefill-decoding interferences', co-optimizes allocation and parallelism per phase, and 'places the two phases according to the serving cluster's bandwidth to minimize the communication caused by disaggregation'. Claims 7.4x more requests or 12.6x tighter SLO vs SOTA systems while staying within latency constraints for >90% of requests. The 'places by cluster bandwidth' sentence is the primary-source statement that KV transfer volume x link bandwidth is the cost model.