inference/research
drafthigh · 0.85updated 2026-10-05

TensorRT-LLM PR #18542: FMHA NVRTC JIT warmup on the cold-start critical path

repo
This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
sources/w7k-trtllm-pr-18542-nvrtc-jit-warmup
Publisher
NVIDIA
Kind
repo
Accessed
2026-10-05
notes
Repo-internal source also cited by ev-trtllm-pr-18542-fmha-jit-warmup, quoted here because it is the single best worked example of NVRTC's real cost in a production engine: at engine startup, TRTLLM-Gen FMHA generation kernels are 'NVRTC-compiled synchronously inside the first warmup forward - ~5-11s per kernel variant depending on host CPU, 80-160s total on MLA fp8-KV configs (DeepSeek-family), ~2min for sparse gen kernels. This sits directly on the engine cold-start critical path.' Vendor-authored, so authoritative about its own cost; the proposed opt-in TRTLLM_GEN_FMHA_ASYNC_WARMUP=1 fix reports internal B200 warmup 'from 163 seconds to approximately 26-30 seconds'. These are runtime codegen costs paid per cold start of a new engine, not C++ build costs - the distinction this repo's build-cost gotcha turns on.

Notes

Repo-internal source also cited by ev-trtllm-pr-18542-fmha-jit-warmup, quoted here because it is the single best worked example of NVRTC's real cost in a production engine: at engine startup, TRTLLM-Gen FMHA generation kernels are 'NVRTC-compiled synchronously inside the first warmup forward - ~5-11s per kernel variant depending on host CPU, 80-160s total on MLA fp8-KV configs (DeepSeek-family), ~2min for sparse gen kernels. This sits directly on the engine cold-start critical path.' Vendor-authored, so authoritative about its own cost; the proposed opt-in TRTLLM_GEN_FMHA_ASYNC_WARMUP=1 fix reports internal B200 warmup 'from 163 seconds to approximately 26-30 seconds'. These are runtime codegen costs paid per cold start of a new engine, not C++ build costs - the distinction this repo's build-cost gotcha turns on.

Cross-references

This record references no other records.