Splitwise: Efficient generative LLM inference using phase splitting (Microsoft)

Characterization source for the two-phase hardware split: 'there are two main phases during an LLM inference request: a compute-intensive prompt computation, and a memory-intensive token generation, each with distinct latency, throughput, memory, and power characteristics', and 'unlike compute-intensive prompt computation phases, token generation phases do not require the compute capability of the latest GPUs, and can be run with lower power and cost'. States the tax and the path: splitting 'requires state transfer from the machine running prompt computation over into the machine generating tokens', 'implemented and optimized using the fast back-plane interconnects available in today's GPU clusters'. Results: 1.4x higher throughput at 20% lower cost, or 2.35x more throughput at same cost and power. This is the source for 'the KV handoff goes over the scale-up backplane, not the NIC'.