18 — Reliability: what actually goes wrong in production serving

The decision this document serves: you have a working deployment. What will break, what will degrade silently, and what should you monitor? The gotcha records in this repo are the evidence base — 47 records of things that went wrong in real deployments, found by people who were not looking for them. The ordering below is by what you can detect, not by severity, because the failures you can detect are the ones you can prevent.

1. The one-paragraph version

Production serving failures fall into three categories, and they have different detection profiles:

Silent wrong output — the server starts, serves every request, emits no error, and returns confident wrong text. These are the most dangerous because throughput monitoring does not catch them. Several are faster when broken, so A/B testing selects for the bug.
Silent performance degradation — the server serves correctly but slower. The cause is usually thermal, power-delivery, or clock-state, and it is invisible unless you monitor the specific signal.
Measurement traps — the deployment is fine but the number you are using to make decisions is wrong. These are the most common and the least recognized.

2. Silent wrong output: the failures throughput monitoring cannot catch

2.1 The pattern

Every record in this section shares one property: the server starts, serves every request, emits no error, no warning and no NaN, and returns confident wrong text or wrong logits. The gotcha records are explicit that these were found only because someone ran a cross-check that a deployment would not run.

2.2 The records, grouped by trigger

Quantization + kernel selection:

NVFP4 models with --dtype bfloat16 produce garbled output on any GPU without native FP4 — blocker. NVFP4 + --dtype bfloat16 on SM < 100 (RTX 4090 SM89, V100 SM70): loads, starts, serves, every response garbled. The FP16 path maps the 5-bit exponent directly; the BF16 path does not.
Setting VLLM_MARLIN_INPUT_DTYPE=fp8 enables Marlin W4A8-FP8 on sm_121a where it silently produces garbage — blocker. sm_121a + WNA16 INT4 MoE + VLLM_MARLIN_INPUT_DTYPE=fp8: coherent-looking repetition loop, and the corrupted path is ~2.5% FASTER end-to-end, so throughput benchmarking selects for the bug.
FlashInfer's b12x MoE backend silently defaults quant_mode to W4A4, so a weight-only NVFP4 checkpoint is computed at the wrong activation precision — major. A checkpoint marked W4A16_NVFP4 served with --moe-backend flashinfer_b12x runs the FP4xFP4 micro-kernel. Silent wrong output. The record says explicitly: "Do not accept a b12x trace showing an FP4 kernel for a W4A16 checkpoint."
compressed-tensors MXFP4 weight-only checkpoints are dispatched to a W4A4 kernel on SM100+, costing ~7.6 average points across AIME/GSM8K/MATH-500 — major. On B200, a compressed-tensors MXFP4 checkpoint with input_activations: null (i.e. W4A16) routes to a W4A4 kernel. Measured cost: 78.16 vs 70.58 on AIME/Math/GSM8K average against the unquantized 82.36 — a 7.58-point accuracy drop caused purely by kernel misrouting.

KV cache + CUDA graphs:

An FP8 KV cache write path in Python is graph-unsafe and silently corrupts attention under CUDA graphs while --enforce-eager looks fine — blocker, silent corruption. 8x A800-80GB (SM80, no native FP8 tensor cores) with --kv-cache-dtype fp8_e5m2, TP=8: the server runs and returns wrong output. The bf16 KV path writes through a fused CUDA op that is graph-safe; the fp8 path goes through plain Python that allocates a fresh tensor and does an advanced-index scatter, which under graph capture corrupts the write.

Speculative decoding + attention:

Disabling FlashInfer Q-quantization with an FP8 KV cache and speculative decoding silently corrupts sliding-window models on SM100 — blocker. 1x B200 (SM100), vLLM 0.23.1rc1, serving a model with sliding-window attention (50 of 60 layers SWA, window 1024) with --kv-cache-dtype fp8_e4m3 and MTP num_speculative_tokens=7. Two independent defects: the spec-as-decode kernel mis-computes sliding-window attention once seq_len > window at certain alignments. Dropping any one of the three flags (spec decode, quantized KV, or the attention config) makes output clean.

Platform-specific:

On GPUs in NVIDIA Confidential Computing mode, vLLM's V2 model runner serves silent garbage — blocker. On an H100 NVL in CC mode inside a TDX guest, vLLM 0.29.0 with the default V2 model runner serves and every response is garbage (token id 0 repeated). VLLM_USE_V2_MODEL_RUNNER=0 fixes it.
On gfx1151 the llama.cpp HIP backend produces corrupt output where Vulkan is correct with byte-identical flags, so cross-backend comparison is the only clean detector — major. On gfx1151 (Strix Halo APU, ROCm 7.2.4), same machine, same build, byte-identical flags: the HIP backend is wrong and Vulkan is right.
llama.cpp HIP on gfx1151 returns wrong-but-plausible logits for any prompt longer than n_ubatch, and the default n_ubatch makes it almost universal — major, silent. Any prompt longer than n_ubatch produces badly wrong logits on HIP/ROCm on gfx1151 — no error, no warning, no crash, and the numbers look plausible. Default n_ubatch is 512.
Multi-GPU HIP on Windows 11 produces fluent gibberish at full speed, and benchmarks do not detect it — blocker. On Windows 11 with a gfx1100 + gfx1201 pair, the HIP backend splits a model across both cards and returns fluent-but-nonsensical word salad with healthy-looking throughput. Building with -DGGML_CUDA_NO_PEER_COPY=ON makes the corruption disappear.
On some AMD boards Vulkan dies with ErrorDeviceLost only when SAM/ReBAR is enabled — blocker. With SAM/ReBAR enabled, every decode on an AMD Radeon AI PRO R9700 fails with vk::Queue::submit: ErrorDeviceLost. Disabling ReBAR fixes the crash but re-introduces the slowdown in Vulkan decode on an AMD dGPU collapses 2-6x when Resizable BAR is disabled in BIOS.

2.3 What to monitor

The records are unanimous that none of these are caught by standard health checks. The monitoring that would catch them:

Output quality spot-checks. Run a fixed prompt with a known-good completion at a fixed interval and compare. Draft-model speculative decoding is not lossless at temperature 0 when the target model is quantized records that greedy sampling with spec decode produces different text on a quantized target — so a greedy-decode spot-check is the cheapest oracle.
Kernel selection logs. FlashInfer's b12x MoE backend silently defaults quant_mode to W4A4, so a weight-only NVFP4 checkpoint is computed at the wrong activation precision and compressed-tensors MXFP4 weight-only checkpoints are dispatched to a W4A4 kernel on SM100+, costing ~7.6 average points across AIME/GSM8K/MATH-500 both produce log lines that name the kernel. Alert on any log line that names a W4A4 kernel for a W4A16 checkpoint.
Perplexity regression. A llama.cpp ROCm commit raised WikiText-2 perplexity from 7.72 to 3024 on a Q4_0 model with no error, and only llama-per perplexity caught it records that llama-perplexity on ROCm/gfx1151 returns a PPL jump of two orders of magnitude. Treat a PPL jump of two orders of magnitude as a hard stop.
Cross-backend comparison. On gfx1151 the llama.cpp HIP backend produces corrupt output where Vulkan is correct with byte-identical flags, so cross-backend comparison is the only clean detector was found by running the same model on two backends. A periodic cross-backend comparison on a fixed prompt is the cheapest detector of backend-specific corruption.

3. Silent performance degradation: the failures health checks cannot catch

3.1 Thermal and power-delivery

HW Thermal Slowdown cuts core clocks by a factor of 2 or more, and nvidia-smi reports it as a microsecond counter rather than a visible failure — major. Sustained-token-rate collapses mid-run with no error, no crash and no assertion. A batch that measured X tokens/s over a short window measures 0.5x over a long window. The GPU is thermally throttling.
Sustained load drives SW Power Cap and thermal throttling, which move the tail of the latency distribution rather than its mean - so a benchmark that reports only p50 hides the regression entirely — major. Mean and median latency look fine while the tail degrades. A deployment passes an SLO defined on p50 and breaks for users. The failures cluster in the long runs — after the node has been at full load for a while, or on the hottest day.
A DGX H100/H200 node continues to run at REDUCED performance with half its PSUs depopulated - the degradation is silent, not an alert — major. After a PSU failure takes three of six PSUs offline, the system "continues to function, but at a reduced performance level." No crash, no error, no degraded-state alarm in normal monitoring. The node serves traffic, passes health checks, and simply delivers less throughput.
HW Power Brake means the PSU asserted: core clocks drop by 2x or more, and the symptom is an unexplained throughput collapse rather than a PSU error — major. nvidia-smi reports 'HW Power Brake' active, meaning an external power brake assertion has been triggered by the system power supply.

What to monitor: nvidia-smi throttle-reason codes (SW Power Cap, HW Thermal Slowdown, HW Power Brake), PSU population per node, and p99 latency over a window long enough to contain a real duty cycle — the MLPerf submissions use min_duration 2400000 ms, and Sustained load drives SW Power Cap and thermal throttling, which move the tail of the latency distribution rather than its mean - so a benchmark that reports only p50 hides the regression entirely notes that this node needs roughly 15 minutes to leave its ramp.

3.2 Clock state

Without locked clocks, cross-part and cross-run comparisons are meaningless - and short bursts measure the parked clock, not the machine — major. A fresh-process matmul canary reads 25.0 TFLOPS in a quiet window and 119.3 TFLOPS minutes later after a heavy workload — on the same machine. Clock parking plus slow ramp hysteresis means short-probe TFLOPS is not a state verdict.

What to monitor: lock clocks with nvidia-smi --lock-gpu-clocks=<min>,<max> for any cross-part or cross-run comparison. Never use short-probe TFLOPS as a state verdict.

3.3 Engine and framework regressions

vLLM 0.30.0's FA4 kernel ignores num_splits on SM90, costing up to 48% decode throughput — major. Decode on H100 (SM90) is 48% slower per token than on v0.26.0 at batch 1 and 36% slower at batch 4, while TTFT is flat. Results stay correct, so nothing fails. The fix is backend selection: attention_backend=TRITON_ATTN recovers ~70%.
FP8 on RDNA3 blows the 600 s engine-ready timeout on first start even though weights load in seconds — major. vLLM on gfx1100 with an FP8 checkpoint aborts during startup on the engine-ready timeout (default 600 s); measured first init was 1474.26 s. Set VLLM_ENGINE_READY_TIMEOUT_S=1800 for the first cold start; once the Triton cache is warm, init drops to ~49 s.
Enabling the V1 engine shrinks usable context several-fold on the same hardware and model — major. Same model and flags: vLLM 0.6.x ran at 12K context on 4x RTX 3070; 0.7.0 with V1 enabled cannot exceed ~3K. Re-tune for the V1 engine rather than carrying V0 values.
vLLM's --gpu-memory-utilization is a fraction of TOTAL VRAM, so KV cache block allocation fails even with memory visibly free — major. "No available memory for the cache blocks." The KV budget is a first-class constraint and the error message names the fix (--gpu_memory_utilization, --max-num-batched-tokens, --max-model-len).
EAGLE/MTP speculative decoding costs 6-10 points of prefix-cache hit rate, and hybrid layouts make the recompute 100x larger — major. Multi-round and repeated-prompt workloads lose ~6-10 points of prefix-cache hit rate when speculative decoding is on. Measure hit rate with and without speculative_config.
Enabling speculative decoding with FlashInfer silently downgrades CUDA graphs to PIECEWISE, costing ~16% decode — major. A single logger.warning announces a CUDA-graph downgrade; decode throughput 47.5 tok/s on FLASHINFER vs 55.2 on FLASH_ATTN (+16%), changing only --attention-backend.
Enabling MTP speculative decoding together with prefix caching silently costs ~20% accuracy — blocker. On a finetuned Qwen3.6 35B-A3B with MTP (num_speculative_tokens=2) plus --enable-prefix-caching, internal classification accuracy drops ~20% versus the identical setup without.

What to monitor: engine version and configuration in the deployment manifest. Any version bump should trigger a re-benchmark and a re-tune of --max-model-len and --max-num-batched-tokens. Enabling the V1 engine shrinks usable context several-fold on the same hardware and model is the clearest example of a version bump silently changing the memory envelope.

4. Measurement traps: the numbers that lie

4.1 Power measurement boundary

GPU board power is 55-59% of system power: CPU, DRAM, NICs, PSU losses and cooling add 41-45% on top, so a joules-per-token figure without its measurement boundary is meaningless — major. A joules-per-token figure is quoted without stating whether it is measured at the GPU package, the accelerator board, the node, or the rack. The number is then compared against another figure measured at a different boundary, and the comparison is wrong by 40-70%. The 'Where Does the Energy Go?' paper profiled 2x RTX PRO 6000 Blackwell and found that GPU-only telemetry misses 41-45% of system energy.
No vendor publishes joules-per-token for inference: the only way to get it is to measure it yourself or estimate from TDP and throughput, and the estimation error is 30-50% because TDP is a thermal ceiling not a power measurement and throughput depends on batch size, sequence length and model — major. The NVIDIA HGX H100/H200 datasheet publishes NO TDP or rack power figure at all. The only available figures are TDP (a thermal design power ceiling, not a measured power draw) and peak throughput (a best-case number at maximum batch size). Estimating joules-per-token from TDP carries 30-50% error.

What to monitor: always state the measurement boundary alongside any joules-per-token figure. If you must estimate system energy from board power, apply a 1.4-1.7x overhead factor depending on the platform.

4.2 Idle power

A fleet at low utilisation can cost MORE in power than the same fleet at high utilisation - the fixed power floor makes naive cost-per-token arithmetic wrong — major. Energy per unit of work is U-shaped in the operating point rather than monotonic, so the cheapest-to-run configuration is not the busiest one. An idle fleet can cost more than a busy fleet.
8xH200 SXM-141GB (MaxQ), MLPerf v4.1 llama2-70b Offline RANGING run: 0 tok/s at 2411.2 W mean node draw - idle floor is 41.7% of the 5806.9 W loaded plateau records that an 8x H200 node draws 2,411 W at idle (zero tokens/s).

What to monitor: utilization, not just throughput. A fleet running at 20% utilization pays 44% of the energy bill for nothing (Modeled US volume-server idle/maximum power dynamic range: 0.44 minimum by 2020 from measured SERT data, 0.28 in the Best Practices scenario, with an explicit ~20-25% idle-power latency floor: idle floor is 0.44 of maximum power for volume servers).

4.3 Carbon intensity

Grid carbon intensity varies 10-50x by region, so a token generated in Quebec (~20 gCO2/kWh) is not the same as one generated in West Virginia (~900 gCO2/kWh) — and almost nobody accounts for it — major. An emissions claim computed as joules-per-token × average grid carbon intensity produces a single number that is wrong by an order of magnitude depending on where the tokens are actually generated. The claim "inference emits X gCO2 per token" is meaningless without the grid region, the carbon intensity value (average or marginal), the time of day, the facility PUE, and the joules-per-token figure with its measurement boundary.

5. The reliability checklist

Before deploying, and at every version bump:

Run a greedy-decode spot-check against a known-good completion. This is the cheapest oracle for silent wrong output (Draft-model speculative decoding is not lossless at temperature 0 when the target model is quantized).
Log the kernel selection and alert on any W4A4 kernel for a W4A16 checkpoint (FlashInfer's b12x MoE backend silently defaults quant_mode to W4A4, so a weight-only NVFP4 checkpoint is computed at the wrong activation precision, compressed-tensors MXFP4 weight-only checkpoints are dispatched to a W4A4 kernel on SM100+, costing ~7.6 average points across AIME/GSM8K/MATH-500).
Monitor throttle-reason codes (nvidia-smi → SW Power Cap, HW Thermal Slowdown, HW Power Brake) and PSU population per node (A DGX H100/H200 node continues to run at REDUCED performance with half its PSUs depopulated - the degradation is silent, not an alert).
Report p99 latency over a 15-minute window, not p50 over a 1-minute window (Sustained load drives SW Power Cap and thermal throttling, which move the tail of the latency distribution rather than its mean - so a benchmark that reports only p50 hides the regression entirely).
Lock clocks for any cross-part comparison (Without locked clocks, cross-part and cross-run comparisons are meaningless - and short bursts measure the parked clock, not the machine).
State the measurement boundary for every joules-per-token figure (GPU board power is 55-59% of system power: CPU, DRAM, NICs, PSU losses and cooling add 41-45% on top, so a joules-per-token figure without its measurement boundary is meaningless).
Re-tune --max-model-len and --max-num-batched-tokens after every engine version bump (Enabling the V1 engine shrinks usable context several-fold on the same hardware and model).
Measure prefix-cache hit rate with and without speculative decoding before enabling spec decode on prefix-heavy workloads (EAGLE/MTP speculative decoding costs 6-10 points of prefix-cache hit rate, and hybrid layouts make the recompute 100x larger).
Set VLLM_ENGINE_READY_TIMEOUT_S=1800 for the first cold start on ROCm (FP8 on RDNA3 blows the 600 s engine-ready timeout on first start even though weights load in seconds).
Do not accept a b12x trace showing an FP4 kernel for a W4A16 checkpoint (FlashInfer's b12x MoE backend silently defaults quant_mode to W4A4, so a weight-only NVFP4 checkpoint is computed at the wrong activation precision).

6. What the records cannot answer

No production-scale reliability study. Every gotcha record is a single report from a single deployment. There is no base rate: the records cannot tell you how likely any of these is in your deployment.
No multi-tenant failure mode. All records are from single-tenant or single-workload deployments. Multi-tenant interference (noisy neighbors, KV cache eviction under memory pressure) is not recorded.
No network failure mode. The records cover GPU, engine, and framework failures but not the network failures (RDMA hangs, InfiniBand congestion, Ethernet packet loss) that dominate multi-node serving reliability.
No persistent-state corruption. All records are from stateless serving. A model that corrupts its own KV cache across requests, or a prefix cache that returns stale entries, would not be caught by any of the monitoring above.