inference/research
verifiedhigh · 0.85majorupdated 2026-10-03

Enabling speculative decoding with FlashInfer silently downgrades CUDA graphs to PIECEWISE, costing ~16% decode

[major] config

Fields

id
gotchas/spec-decode-silently-downgrades-cudagraphs
Class
config
Severity
major
Concepts
qwen3-5-122b-a10b-int4
Symptom
A single logger.warning announces that cudagraph_mode was downgraded; decode throughput is 47.5 tok/s on FLASHINFER vs 55.2 tok/s on FLASH_ATTN (+16%), changing only --attention-backend. The warning names neither the cost nor the alternative, so it reads as an informational line.
Root cause
With speculative decoding enabled (MTP/EAGLE, num_speculative_tokens >= 1) and the FlashInfer backend on its native non-trtllm-gen decode path, vLLM silently downgrades cudagraph_mode from FULL_AND_PIECEWISE to PIECEWISE. The decode step then pays per-step kernel-launch overhead that a full decode graph would elide. Measured on GB10 / DGX Spark (sm_121a, unified LPDDR5X, ~273 GB/s) - bandwidth-bound, so launch overhead is unusually visible; trtllm-gen is not usable on sm_121.
Workaround
Use FLASH_ATTN (FA2/FA3) instead of FLASHINFER with speculative decoding to keep the full decode graph, or drop speculative decoding. Check the resolved cudagraph_mode in the startup log rather than trusting the flag you passed.
notes
The general lesson is the measurement trap, not this specific backend: a performance optimization silently disabled by another optimization, announced only by a warning that does not quantify the cost. Same shape as gotcha-mtp-plus-prefix-caching-accuracy-loss but on the speed axis instead of the accuracy axis.

Notes

The general lesson is the measurement trap, not this specific backend: a performance optimization silently disabled by another optimization, announced only by a warning that does not quantify the cost. Same shape as gotcha-mtp-plus-prefix-caching-accuracy-loss but on the speed axis instead of the accuracy axis.

Cross-references