LLM Inference Performance Engineering: Best Practices for Tuning Decoding

Benchmark blog from a production inference team. Cited for two things: (a) the four serving metrics and the explicit statement that throughput (aggregate across all users) trades directly against time-per-output-token (per user), and (b) Table 2, a static-batching MPT-7B throughput sweep with batch size 1..128 on 1x/2x/4x A100 and 1x/2x A10 at fixed 512 input / 64 output tokens. All numbers are MosaicML's own measurements on their own serving stack, so measured_by for anything taken from it is 'vendor' for Databricks, not an independent third party.