TensorRT-LLM per-GPU-architecture quantization support matrix (which format has a kernel on which arch)

GROUP SIZE LEFT NULL 2026-10-04 and NOT MEANINGFUL FOR THIS RECORD: it spans eleven format columns whose group sizes are 16, 32 and 128, so no single value describes it. Read the per-format records for group sizes. PROXY ENUM VALUE - DECLARED 2026-10-04: this is a MATRIX record spanning eleven format columns (NVFP4, MXFP4, FP8 per-tensor/block/rowwise, FP8 KV, NVFP4 KV, AWQ and GPTQ at W4A8/W4A16), filed under 'nvfp4'. NVFP4 is a defensible anchor - it is the column that changes availability most across architectures and the reason the sm120 row matters - but it is one of eleven, and a query for NVFP4 returns this matrix rather than the format record. A reader wanting NVFP4 itself should use the nvfp4 record. REQUESTED ENUM ADDITION: a 'hardware-matrix' or 'support-matrix' value distinct from any format. THE FORMAT-vs-KERNEL DISTINCTION, expressed as a matrix. A format being IMPLEMENTED in a library is not the same as a kernel existing for your GPU, and this table is the vendor stating which of its own formats have kernels per architecture. Columns: NVFP4, MXFP4, FP8(per tensor), FP8(block scaling), FP8(rowwise), FP8 KV Cache, NVFP4 KV Cache, W4A8 AWQ, W4A16 AWQ, W4A8 GPTQ, W4A16 GPTQ. Rows verbatim: Blackwell(sm120) = NVFP4 Y, MXFP4 Y, FP8(per tensor) Y, FP8(block scaling) '.', FP8(rowwise) '.', FP8 KV Y, NVFP4 KV '.', ALL FOUR AWQ/GPTQ '.'. Blackwell(sm100/103) = NVFP4 Y, MXFP4 Y, FP8(per tensor) Y, FP8(block scaling) Y, FP8(rowwise) '.', FP8 KV Y, NVFP4 KV Y, W4A8 AWQ Y, W4A16 AWQ Y, W4A8 GPTQ Y, W4A16 GPTQ Y. Rubin(sm107) = identical to sm100/103 EXCEPT NVFP4 KV Cache '.'. Hopper = NVFP4 '.', MXFP4 '.', FP8(per tensor) Y, FP8(block scaling) Y, FP8(rowwise) Y, FP8 KV Y, NVFP4 KV '.', W4A8 AWQ Y, W4A16 AWQ Y, W4A8 GPTQ Y, W4A16 GPTQ Y. Ada Lovelace = NVFP4 '.', MXFP4 '.', FP8(per tensor) Y, FP8(block scaling) '.', FP8(rowwise) '.', FP8 KV Y, NVFP4 KV '.', W4A8 AWQ Y, W4A16 AWQ Y, W4A8 GPTQ Y, W4A16 GPTQ Y. Ampere = NVFP4 '.', MXFP4 '.', FP8(per tensor) '.', FP8(block scaling) '.', FP8(rowwise) '.', FP8 KV Y, NVFP4 KV '.', W4A8 AWQ '.', W4A16 AWQ Y, W4A8 GPTQ '.', W4A16 GPTQ Y. THIS CONFIRMS, INDEPENDENTLY OF THE FORUM REPORTS ALREADY IN THIS REPO, THE SM120 GAP. The already-recorded finding was that sm120 has no FP8 block-scaled GEMM; the vendor table states the same thing as a '.' in the FP8(block scaling) column for Blackwell(sm120) while sm100/103 and Rubin(sm107) both carry Y. Extend it with three facts this adds that the existing records did not have. (1) sm120 ALSO lacks the NVFP4 KV cache path, which sm100/103 has. (2) sm120 lacks ALL FOUR AWQ and GPTQ cells - so on an RTX PRO 6000 Blackwell or a 5090, TRT-LLM can serve NVFP4, MXFP4 and per-tensor FP8 but none of the 4-bit weight-only formats, which is the opposite of the Ampere/Ada pattern where AWQ/GPTQ W4A16 are Y. (3) FP8(rowwise) is ONLY on Hopper - it is '.' on Blackwell sm120, sm100/103, sm107, Ada AND Ampere, which makes rowwise FP8 the narrowest-supported FP8 variant in the table despite being the one that needs no per-tensor calibration. THE RECIPE IS ARCH-DEPENDENT, WHICH IS THE SUBTLEST LINE IN THE DOC: 'FP8 block wise scaling GEMM kernels for sm100/103/107 are using MXFP8 recipe (E4M3 act/weight and UE8M0 act/weight scale), which is slightly different from SM90 FP8 recipe (E4M3 act/weight and FP32 act/weight scale).' So 'FP8 block scaling' is one COLUMN covering at least two numerically different scale formats depending on architecture. A reader who computes memory or quality for block-scaled FP8 and assumes the SM90 recipe will be wrong on Blackwell in a way the table does not flag as an error. CAVEAT ON THE SM120 ROW: this row is about TensorRT-LLM's kernels specifically. It is consistent with the separate repo gotcha that TRT-LLM lists the SM120 RTX PRO 6000 as 'N/A - Doesn't support the arch' outright, and with the vLLM-side reports that cutlass does not support arch=120a. Where a format is '.' here it means TRT-LLM ships no kernel for it, NOT that no engine can serve it - vLLM and SGLang serve AWQ/GPTQ W4A16 on Blackwell via Marlin and Machete, which is a different kernel set. Read this table as 'what TensorRT-LLM can do', never as 'what this GPU can do'. === RE-VERIFIED CELL BY CELL 2026-10-04, ALL 72 CELLS CONFIRMED AGAINST THE LIVE PAGE. === The Hardware Support Matrix table was re-fetched from the vendor documentation on 2026-10-04 and every one of the six rows by eleven columns matches this record's transcription exactly, including the awkward cells: Blackwell(sm120) = NVFP4 Y, MXFP4 Y, FP8(per tensor) Y, FP8(block scaling) ., FP8(rowwise) ., FP8 KV Y, NVFP4 KV ., and all four AWQ/GPTQ cells .; Blackwell(sm100/103) = Y Y Y Y . Y Y Y Y Y Y; Rubin(sm107) = identical to sm100/103 except NVFP4 KV Cache .; Hopper = . . Y Y Y Y . Y Y Y Y; Ada Lovelace = . . Y . . Y . Y Y Y Y; Ampere = . . . . . Y . . Y . Y. The MXFP8-recipe footnote is also confirmed verbatim. NO CELL IN THIS MATRIX WAS FOUND WRONG, which is worth stating as a finding in its own right - it is the control case against the vLLM matrix, where one cell did not survive. Status stays verified and confidence rises to 0.95: this is now two independent reads of the vendor table on different dates that agree exactly, plus the corroborating cross-engine checks below. THE THREE CROSS-ENGINE DISAGREEMENTS THIS TABLE CREATES, NOW MEASURED RATHER THAN ASSUMED. Because vLLM's kernel source was re-derived in the same pass, each TensorRT-LLM '.' can be labelled by what actually happens on that part in the other engine, and this is the layer the previous version of this record left implicit. (1) MXFP4 ON sm120: TRT-LLM says Y UNCONDITIONALLY, vLLM says YES-BUT-ONLY-IF-AN-OPTIONAL-MODULE-IS-INSTALLED. vLLM's CUDA MXFP4 candidate list is [FlashInferMxFp4, MarlinMxFp4, HummingMxFp4, B12xMxFp4, EmulationMxfp4], and the gate that decides it is FlashInferMxFp4LinearKernel.is_supported() = 'has_device_capability(100) and has_flashinfer_cutedsl()'. Because has_device_capability is a >= comparison, 120 >= 100, so the architecture half of that gate PASSES on sm120 - the only thing standing between an RTX PRO 6000 Blackwell or a 5090 and a native MXFP4 GEMM in vLLM main is whether the FlashInfer CuTe-DSL Python module is installed. Its failure string, 'FlashInfer + >=sm_100 (Blackwell) required', is itself misleading because the sm_100 half is already satisfied. B12xMxFp4LinearKernel is the other sm120 route and needs pip install vllm[b12x]. MarlinMxFp4 is weight-only and logs that it ignores the requested activation quantization. EmulationMxfp4LinearKernel.is_supported() returns (True, None) unconditionally and logs 'The current platform does not support native MXFP4 computation. Using EmulationMxfp4LinearKernel, with simulated weight dequantization and activation QDQ ... with the linear layers computed in high precision.' So the honest cross-engine statement is a three-way split, not a two-way one: sm100/103 native in vLLM with FlashInfer installed, sm120/121 native in vLLM only if one of two optional packages is present, sm90 and below emulated unconditionally in vLLM. On a stock install the same RTX 5090 that TensorRT-LLM serves MXFP4 natively will serve it in software emulation under vLLM - the model loads, the output is correct, and the only evidence is a one-line warning at startup. That is the single most decision-relevant cell in this record and it was previously a bare Y. Note the useful asymmetry for a reader: this class of gap is FIXABLE by an install, unlike the NVFP4-on-Hopper gap in case (b), which is a silicon fact. (2) NVFP4 KV CACHE ON sm107: TRT-LLM says '.', vLLM says supported. supports_trtllm_attention() returns current_platform.is_device_capability_family(100) after special-casing sm90 and sm12x, and that family test is cc//10 == 100//10, so it is true for cc 100 through 109 - which includes sm107. So Rubin's NVFP4 KV cache is a vLLM capability that TensorRT-LLM has not shipped, not a hardware gap. (3) FP8 BLOCK SCALING ON sm120: TRT-LLM says '.', and vLLM's CUTLASS path agrees - cutlass_scaled_mm_supports_block_fp8 returns false below SM90, demands CUDA >= 12.0 on sm90 and CUDA >= 12.8 on cc>=100, with NO sm120 branch, so block-scaled FP8 via CUTLASS is unavailable on sm120 in vLLM too. But vLLM's C++ tree does carry per-architecture blockwise FP8 CUTLASS translation units named scaled_mm_blockwise_sm90_fp8, _sm100_fp8 and _sm120_fp8, so a dedicated sm120 blockwise kernel exists in the source tree even though the support predicate declines to report it. Treat sm120 block-scaled FP8 as contested between engines rather than settled, and check the startup log. THE VERSION GATE THAT DOES NOT FIT IN A Y/. TABLE, because it is not architectural. cutlass_scaled_mm_supports_fp4 requires the CUDA RUNTIME to be >= 12.8 in addition to cc being in [100,120) or [120,130); cutlass_scaled_mm_supports_block_fp8 requires CUDA >= 12.0 on sm90 and >= 12.8 on cc>=100; cutlass_scaled_mm_supports_fp8 requires CUDA >= 12.4 on sm89 and >= 12.0 on sm90. So every Blackwell cell in this matrix is additionally a CUDA-version claim: the vendor Y is real but conditional on a toolkit version the matrix does not name. The C++ comment explains why the predicate is version-aware at all - 'Only report support when the SM-specific kernel was actually compiled in, so the Python-side backend selector does not choose CUTLASS and then hit TORCH_CHECK_NOT_IMPLEMENTED (or worse, fall through to Marlin).' WHAT A '.' IN THIS TABLE NOW MEANS PRECISELY, THREE DIFFERENT THINGS. It is not one thing, and conflating them is how a capacity plan goes wrong. (a) 'The engine has no kernel and will refuse' - AWQ/GPTQ on sm120, W4A8 on Ampere. The request fails loudly; nothing is silently slow. (b) 'The engine has no kernel but another path serves it at lower precision' - NVFP4 on Ada/Ampere/Hopper via Marlin W4A16, the already-recorded silent degradation. (c) 'The engine has a kernel and a DIFFERENT engine does not' - MXFP4 on sm120, NVFP4 KV on sm107. Read case (c) as an engine choice, not a hardware limit. Every '.' in this record should be assigned one of these three before it is quoted in a plan.