Canarying an inference service is harder than canarying a web service because a request's cost is not knowable in advance and the comparison is non-deterministic, so there is no clean 'new version is worse' signal

A canary at 5% traffic produces no usable verdict. Error-rate comparison is meaningless because both versions return 200; latency comparison is confounded because a different replica mix behind the canary has a different queue depth and therefore different latency for reasons unrelated to the change; and output-quality comparison is confounded by sampling non-determinism, so a diff between versions mostly measures randomness. Teams end up either shipping the canary on vibes or reverting to percentage-based rollout of image tags with no feedback loop at all.

Four compounding reasons. (1) Cost is not knowable before serving: a request's resource consumption 'is driven by long input contexts and the unpredictable autoregressive decode loop (output tokens)', so the same request rate against two versions is not the same load, and traffic-splitting by percentage does not split by cost. (2) The comparison is non-deterministic: sampled output varies run to run, so a byte-diff verdict is noise. (3) Disaggregated serving adds a compatibility constraint that makes same-pool canarying unsafe: NIXL 'pulls the KVs directly from the GPU KV cache memory of the remote instance', and 'vLLM has multiple attention kernel implementations, each of which can have slightly different KV cache layouts', so mixing versions in one pool can produce wrong results rather than merely different ones. (4) The router currently cannot be trusted to keep the versions apart even where it tries: llm-d warns 'The llm-d EPP currently assumes all P and D instances within an InferencePool are compatible and will therefore schedule requests to any arbitrary pair of P and D instances'.

Where the routing layer does support version separation, use it: sgl-router's --pd-version-group-label pairs a prefill worker only with decode workers sharing that EndpointSlice label 'so KV never crosses versions', which requires one Service per role and version, and unlabeled workers form their own group. For a disaggregated upgrade, llm-d's recommendation is structural rather than statistical — 'it is currently recommended to create a new InferencePool for upgrading model servers. When deploying with a Gateway, traffic can be gradually shifted to the new InferencePool by modifying the HTTPRoute', which makes the canary a route shift you can control precisely rather than a percentage you hope means something. SGLang exposes a related guard at the fleet level with --override-sampling-params and --sampling-param-conflict, which pins the sampling configuration for every client of the router so the two versions cannot drift on temperature/top_p/top_k/min_p/repetition_penalty — necessary because those five are resolved from the model's own generation_config and 'which is what makes them drift when a deployed image changes'. If you canary on anything, canary on latency SLO compliance at matched load (llm-d's saturation signal measures P90 TTFT and TPOT against SLO, which is exactly the comparison a percentage split can support), and pin sampling first.

Genuinely unsolved: there is no output-quality regression signal in the routing layer. Every mechanism above addresses safety (don't mix incompatible versions) or fairness (make the traffic split meaningful), but deciding whether the new model produces better text at matched load is a separate evaluation problem that serving infrastructure does not solve. Note also that version skew within a single prefix-cache-affecting deployment has a second cost: a replica that boots cold routes cache-blind for the whole fleet once per replica on every rolling update, so every rollout silently degrades locality regardless of the canary question. SCHEME RECLASSIFIED (2026-10-04): was 'config'.