inference/research
verifiedhigh · 0.88majorupdated 2026-10-04

Average tokens/sec is not an SLO for LLM serving and will not detect the failure users actually report; the two that matter are TTFT and TPOT

[major] measurement

Fields

id
gotchas/obs-serving-slos-ttft-tpot-not-tok-s
Class
measurement
Severity
major
Concepts
slottfttpotitlthroughputcapacity-planningdashboardalerting
Symptom
A dashboard is green because output throughput meets target while users report the service 'feels stuck'. Raising concurrency to hit the throughput target makes TTFT worse without moving the throughput number off target in a way anyone notices. On the other side, a capacity review quotes a single tok/s figure from a benchmark, and three months later the fleet cannot serve the traffic because that figure was measured at batch size 1 with no prefill.
Root cause
Throughput is a capacity number, not an experience number, and the two can move in opposite directions because a serving engine trades one for the other. Throughput is aggregate over concurrent requests; a user experiences a queue wait (invisible in throughput), a prefill, then a stream of inter-token gaps. Two specific mechanisms make this concrete. Preemption: the scheduler frees a running request's KV blocks, the step still runs and still emits tokens, so throughput is unaffected while that user's latency is destroyed (see obs-preemption-hides-latitude-from-throughput). Admission: queuing requests cost nothing measurable in throughput and everything in TTFT - vLLM separates `vllm:num_requests_waiting{reason="capacity"}` from other deferrals precisely so you can tell which is happening. There is also a unit trap underneath the SLO choice: average tok/s has no definition without a concurrency level and an input/output length distribution, so it cannot be an SLO in the way a latency percentile can - a percentile is a statement about a fraction of users, whereas a mean throughput is a statement about the whole fleet. This is also why the engines' own guidance points at latency: vLLM's design doc classifies request-level metrics as 'often the SLOs that an SRE monitoring vLLM will be tracking' and frames server-level metrics as the explanation for them - 'The mental model is that server-level metrics help explain the values of request-level metrics.'
Workaround
Two latency SLOs, stated as percentiles over per-request histograms, plus throughput as a capacity constraint rather than a target. TTFT because it is what a user perceives before any output appears and it is where queueing, prefill and detokenization all land (SGLang's docs are precise that for streaming requests TTFT ends when the first output arrives after detokenization, 'which may contain no printable text' - a detokenizer stall is a TTFT regression). TPOT because it is the rate at which a response streams once started. Serve them from `vllm:time_to_first_token_seconds` and `vllm:request_time_per_output_token_seconds` (or the sglang:/lmdeploy:/trtllm: equivalents), keep ITL alongside for stall detection only, and treat queue depth plus KV usage as the leading indicators that explain a breach. Critically, override the histogram buckets for any sub-second target: vLLM's default `request_latency` family bottoms out at 0.3 s, which cannot resolve a 200 ms SLO, so use --custom-histogram-buckets - and remember each added boundary costs one extra time series per metric per label set. Also do not set the SLO on an engine that cannot measure it: TGI has no TTFT metric, llama.cpp has no latency metric at all, and Ollama has neither.
notes
The dashboards people actually run are a useful tell for which metrics have proven their worth. vLLM ships examples/observability/prometheus_grafana/grafana.json whose panels query, at p50/p90/p95/p99: e2e_request_latency_seconds, inter_token_latency_seconds, request_time_per_output_token_seconds and time_to_first_token_seconds, plus rate() panels on prompt_tokens_total and generation_tokens_total, gauges on num_requests_running and num_requests_waiting, kv_cache_usage_perc, prompt- and generation-token histograms, request_success by finish_reason, and rate() on request_queue_time_seconds, request_prefill_time_seconds, request_decode_time_seconds and request_max_num_generation_tokens. SGLang's shipped dashboard (examples/monitoring/grafana/dashboards/json/sglang-dashboard.json) is narrower: e2e_request_latency_seconds and time_to_first_token_seconds at p50/p90/p99, plus gauges sglang_num_running_reqs, sglang_gen_throughput, sglang_cache_hit_rate and sglang_num_queue_reqs. LMDeploy's dashboard mirrors vLLM's almost exactly (same p50/p90/p95/p99 panels plus gpu_cache_usage_perc), which is what you would expect from the same code ancestry. The pattern in all three: latency percentiles lead, throughput counters support, and queue/KV gauges explain. Nobody ships a tok/s-only dashboard. Also worth knowing for the queue-time SLO: a request that never gets admitted still increments your queue-time histogram only if the engine records the wait for completed requests - vLLM's `vllm:request_queue_time_seconds` is 'Histogram of time spent in WAITING phase for request' recorded on completion, so a backlog of abandoned or timed-out requests is invisible to it and shows up only in the gauge.

Notes

The dashboards people actually run are a useful tell for which metrics have proven their worth. vLLM ships examples/observability/prometheus_grafana/grafana.json whose panels query, at p50/p90/p95/p99: e2e_request_latency_seconds, inter_token_latency_seconds, request_time_per_output_token_seconds and time_to_first_token_seconds, plus rate() panels on prompt_tokens_total and generation_tokens_total, gauges on num_requests_running and num_requests_waiting, kv_cache_usage_perc, prompt- and generation-token histograms, request_success by finish_reason, and rate() on request_queue_time_seconds, request_prefill_time_seconds, request_decode_time_seconds and request_max_num_generation_tokens. SGLang's shipped dashboard (examples/monitoring/grafana/dashboards/json/sglang-dashboard.json) is narrower: e2e_request_latency_seconds and time_to_first_token_seconds at p50/p90/p99, plus gauges sglang_num_running_reqs, sglang_gen_throughput, sglang_cache_hit_rate and sglang_num_queue_reqs. LMDeploy's dashboard mirrors vLLM's almost exactly (same p50/p90/p95/p99 panels plus gpu_cache_usage_perc), which is what you would expect from the same code ancestry. The pattern in all three: latency percentiles lead, throughput counters support, and queue/KV gauges explain. Nobody ships a tok/s-only dashboard. Also worth knowing for the queue-time SLO: a request that never gets admitted still increments your queue-time histogram only if the engine records the wait for completed requests - vLLM's `vllm:request_queue_time_seconds` is 'Histogram of time spent in WAITING phase for request' recorded on completion, so a backlog of abandoned or timed-out requests is invisible to it and shows up only in the gauge.

Cross-references

Affects (16)