TensorRT-LLM PR #18542: FMHA NVRTC JIT warmup on the cold-start critical path
Fields
sources/w7k-trtllm-pr-18542-nvrtc-jit-warmupNotes
Repo-internal source also cited by ev-trtllm-pr-18542-fmha-jit-warmup, quoted here because it is the single best worked example of NVRTC's real cost in a production engine: at engine startup, TRTLLM-Gen FMHA generation kernels are 'NVRTC-compiled synchronously inside the first warmup forward - ~5-11s per kernel variant depending on host CPU, 80-160s total on MLA fp8-KV configs (DeepSeek-family), ~2min for sparse gen kernels. This sits directly on the engine cold-start critical path.' Vendor-authored, so authoritative about its own cost; the proposed opt-in TRTLLM_GEN_FMHA_ASYNC_WARMUP=1 fix reports internal B200 warmup 'from 163 seconds to approximately 26-30 seconds'. These are runtime codegen costs paid per cold start of a new engine, not C++ build costs - the distinction this repo's build-cost gotcha turns on.
Cross-references
This record references no other records.