Quantization runtime-version gates per format per architecture (CUDA / FlashInfer / torch / TVM-build / optional-package), the axis no support matrix has a column for

PROXY ENUM VALUE - DECLARED 2026-10-05: a MATRIX record filed under 'mxfp8'; use the per-format records for the formats themselves. SIBLING OF w6q-kernel-matrix-native-vs-emulated, and the two are meant to be read together: that record says WHICH KERNEL runs, this one says what RUNTIME has to be true for that kernel to exist. Read here first when a format is 'supported' on the right hardware and still fails or silently degrades. THE CENTRAL POINT. A support matrix has one axis — the architecture. Real quantized serving has two: the architecture AND the runtime. The runtime axis is per-format, per-arch, and it is invisible in every Y/./blank table in the repo. Worse, the two failure modes are different and only one of them tells you: a RUNTIME-TOO-OLD gate usually produces a clean fallback or a clean error; a BUILD-WITHOUT-THE-KERNEL gate produces silence. Read the last section of this record before you trust any cell. ---- CUDA RUNTIME GATES, from vLLM's C++ (re-read 2026-10-05 from main @ 528772a4; all four re-confirmed unchanged from wave 4 except the fourth, which is new) ---- FP8 per-tensor (CUTLASS). cutlass_scaled_mm_supports_fp8(cc): CUDA >= 12040 for 89 <= cc < 90; CUDA >= 12000 for cc >= 90; false otherwise. In-code comment: 'CUTLASS FP8 kernels need at least CUDA 12.0 on SM90 systems (Hopper) and CUDA 12.4 on SM89 systems (Lovelace)'. THE SHARP BIT: there is no branch above cc>=90, so sm100, sm103, sm107 and sm120 all take the Hopper rule. Above Hopper, FP8 per-tensor is CUDA-version-gated, NOT arch-gated. Practical consequence: an FP8 W8A8 deployment on Blackwell or on Ada needs CUDA 12.4 or newer even though both are 'newer than the Hopper that needs 12.0'. FP8 block-scaled (CUTLASS). cutlass_scaled_mm_supports_block_fp8(cc): CUDA >= 12080 for cc >= 100; CUDA >= 12000 for 90 <= cc < 100; false otherwise. Comment: 'CUTLASS block-quantized FP8 kernels need at least CUDA 12.0 and at least SM90 (Hopper)'. So the DeepSeek-style 1x128/128x128 layout needs 12.0 on Hopper and 12.8 on Blackwell — the SAME Blackwell part needs a half-major-version newer CUDA for block-scaled FP8 than for per-tensor FP8. That is a 0.8-version spread on one device and no vendor table expresses it. NVFP4 (CUTLASS). cutlass_scaled_mm_supports_fp4(cc): calls cudaRuntimeGetVersion() and returns false unless runtimeVersion >= 12080 — a RUNTIME query, not the compile-time CUDA_VERSION macro the other two use; then true only for 100 <= cc < 120 when ENABLE_NVFP4_SM100 compiled in, or 120 <= cc < 130 when ENABLE_NVFP4_SM120 compiled in; else false. Comment: 'Only report support when the SM-specific kernel was actually compiled in, so the Python-side backend selector does not choose CUTLASS and then hit TORCH_CHECK_NOT_IMPLEMENTED (or worse, fall through to Marlin).' The failure it prevents is named in the code, and it is the same failure this whole record is about. CUTLASS grouped FP8 MoE — NEW, AND PREVIOUSLY UNRECORDED. cutlass_group_gemm_supported(cc): 'CUTLASS grouped FP8 kernels need at least CUDA 12.3 and SM90 (Hopper) or CUDA 12.8 and SM100 (Blackwell). Only report archs that have an actual cutlass_moe_mm dispatch compiled into this file.' Returns CUDA >= 12080 for 100 <= cc < 120 under ENABLE_CUTLASS_MOE_SM100; CUDA >= 12030 for 90 <= cc < 100 under ENABLE_CUTLASS_MOE_SM90; else false. THE OPERATIONALLY IMPORTANT CONSEQUENCE: dense and MoE have DIFFERENT CUDA FLOORS ON THE SAME GPU. A Hopper part that clears the FP8 per-tensor gate at CUDA 12.0 can still fail the FP8-MoE gate at CUDA 12.3. If you are sizing a Hopper MoE deployment, 12.0 is the wrong number to quote from the FP8 rows of any matrix; 12.3 is. And 12.8 on Blackwell, same as NVFP4. THE FOUR-GATE SUMMARY FOR A SINGLE BLACKWELL B200 UNDER CUDA 12.6: FP8 per-tensor YES (>=12.0 branch). FP8 block-scaled NO (needs 12.8). FP8 grouped MoE NO (needs 12.8). NVFP4 NO (needs 12.8 runtime). Under CUDA 12.8: all four YES. One minor CUDA bump moves three of the four native GEMM families on that part. There is no field in any schema in this repo for that. ---- SGLANG'S OWN ARCH PREDICATES ARE VERSION GATES, NOT SILICON TESTS ---- python/sglang/srt/utils/common.py, built on _check_cuda_device_version(device_capability_majors=[...], cuda_version=(M, N)): is_blackwell (aliases is_blackwell_supported) = majors [10, 11, 12] AND CUDA >= (12, 8); is_sm100_supported = majors [10] AND >= (12, 8); is_sm120_supported = majors [12] AND >= (12, 8); is_sm100_or_sm110_supported = majors [10, 11] AND >= (12, 8) ('Datacenter Blackwell (SM100) plus SM110; excludes consumer Blackwell (SM120)'); is_sm90_supported = majors [9] AND >= (12, 3); is_sm80_supported = majors [8] AND >= (11, 0); is_ampere_with_cuda_12_3 = majors [8] AND >= (12, 3); is_hopper_with_cuda_12_3 = majors [9] AND >= (12, 3). Plus two EXACT-MATCH predicates that the major-based helper structurally cannot express, with the reason in a comment: 'GB10 (DGX Spark and OEM equivalents). Not expressible with _check_cuda_device_version, which only matches on the major.' — is_sm120() == (12,0) exactly, is_sm121() == (12,1) exactly. CONSEQUENCE: 'is Blackwell' in SGLang means CUDA >= 12.8 AND Blackwell, so the same B200 takes a different branch under CUDA 12.6 than under 12.8, and an SM121 GB10 is invisible to every majors-based predicate. SGLang's cutlass_fp8_supported() matches vLLM's C++ exactly (major >= 9 AND CUDA >= (12,0); major==8 and minor==9 AND CUDA >= (12,4); else False), and its FlashInfer per-tensor FP8 floor is a three-arch list, not a version: is_flashinfer_available() and (is_sm90 or is_sm100 or is_sm120). ---- FLASHINFER VERSION GATES, NAMED IN THE CODE ---- FlashInfer >= 0.6.11, by PR number. SGLang: RuntimeError('moe_runner_backend=flashinfer_mxfp4 on SM90 requires the interleave_moe_{weights,scales}_for_sm90_mixed_gemm helpers from FlashInfer PR #3084 (>= 0.6.11). Upgrade flashinfer-python or pick a different backend (e.g. marlin / triton_kernel).') — a version gate on the MXFP4 MoE path, and the error names the alternative backends, which makes it one of the better-behaved gates in the stack. A RENAME, which is the same breakage with a worse error message. vLLM has_flashinfer_b12x_gemm() accepts either Sm120B12xBlockScaledDenseGemmKernel or Sm120BlockScaledDenseGemmKernel, commented 'FlashInfer 0.6.11 renamed Sm120BlockScaledDenseGemmKernel -> Sm120B12xBlockScaledDenseGemmKernel (commit 223f2a49). Accept either.' The consequence of pinning the wrong FlashInfer minor is that every b12x kernel reports False — and those are the sm120 native NVFP4, MXFP4, MXFP8 and FP8-block paths — so the selector drops to Marlin W4A16 and says nothing about FlashInfer. MODULE-PRESENCE gates, which are optional-install gates in disguise: has_flashinfer_cutedsl() = has_flashinfer() and importlib.util.find_spec('flashinfer.cute_dsl') is not None. It gates FlashInferCuteDslNvFp4LinearKernel, FlashInferCuteDslNvFp4W4A16LinearKernel, FlashInferMxFp4LinearKernel, FlashInferCutedslMxfp8LinearKernel, and transitively has_flashinfer_bf16_fp4(), has_flashinfer_b12x_gemm() and has_flashinfer_cutedsl_grouped_gemm_nt_masked(). has_flashinfer_bf16_fp4() further requires flashinfer.gemm to expose BOTH mm_bf16_fp4 and prepare_bf16_fp4_weights. has_flashinfer_fp8_blockscale_gemm() requires is_device_capability(90) EXACTLY plus the attribute flashinfer.gemm.fp8_blockscale_gemm_sm90. ENV-VAR gate on top of all of that: is_flashinfer_fp8_blockscale_gemm_supported() = VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER AND has_flashinfer_fp8_blockscale_gemm(). Block-FP8 GEMM on Hopper is switchable off by configuration, with no architecture or version involved. ---- TORCH GATES ---- SGLang NVFP4 KV: requires hasattr(torch, 'float4_e2m1fn_x2'), else ValueError('--kv-cache-dtype=nvfp4 requires torch.float4_e2m1fn_x2 support. Please use PyTorch 2.8.0+ with CUDA 12.8+.'). The message bundles a torch floor and a CUDA floor in one string, which is why it is worth quoting whole. SGLang also emits a WARNING('%s KV Cache might lead to an accuracy drop!') at request time for the same dtype — a version gate and an accuracy caveat in the same branch. vLLM NVFP4 GEMM, via TorchNvFp4LinearKernel: requires is_device_capability_family(100), hasattr(torch, '_scaled_mm') and hasattr(torch, 'float4_e2m1fn_x2'). Same two torch attributes, expressed as capability gates with reasons 'torch._scaled_mm not available' / 'torch.float4_e2m1fn_x2 not available'. vLLM NVFP4 KV: additionally requires force_use_trtllm_attention() and supports_trtllm_attention() true for BOTH prefill and decode, else ValueError(f'--kv-cache-dtype {self.cache_dtype} requires the SM100 trtllm-gen FlashInfer path.'), and is disabled under VLLM_BATCH_INVARIANT. vLLM block-FP8 torch rung is TORCH-ARCH-GATED, not just vLLM-arch-gated: BlockWiseTorchFP8ScaledMMLinearKernel's comment says 'torch._scaled_mm DeepSeek-style (1x128, 128x128) block scaling is only implemented for SM90 (Hopper) on NVIDIA CUDA. See PyTorch's _check_deepseek_support()', and it returns False for anything not exactly SM90. So the generic torch rung of the block-FP8 ladder is narrower than the CUTLASS rung directly above it. ---- BUILD-TIME AND OPTIONAL-PACKAGE GATES: THE ONES THAT FAIL SILENTLY ---- These are the dangerous class, because the Python predicate returns False with no error, the selector takes the next rung, and the next rung is usually a weight-only kernel that produces correct output. ENABLE_NVFP4_SM100 / ENABLE_NVFP4_SM120 decide whether native NVFP4 CUTLASS exists at all. ENABLE_SCALED_MM_SM90 / SM100 / SM120 / C2X decide the FP8 arch dispatch — note that cutlass_scaled_mm() itself will happily dispatch an sm80 or sm75 kernel, so the ARCH list below the predicate is strictly WIDER than the predicate; that gap is exactly why Ampere FP8 reaches Marlin rather than a compiled-but-unadvertised CUTLASS FP8 kernel. ENABLE_CUTLASS_MOE_SM90 / SM100 gate grouped FP8 MoE. 'pip install vllm[b12x]' gates every B12x kernel, each ending in 'Install the B12X backend with `pip install vllm[b12x]`' — and B12x is the sm120 native route for NVFP4, MXFP4, MXFP8 and block-FP8, so a stock install loses all four at once. The Humming family (HummingNvFp4, HummingMxFp4, HummingMxFp6, HummingMxfp8, HummingFP8, HummingInt8, HummingLinear) all gate on 'Humming is not installed' at SM75+. MLC-LLM's cutlass groupwise GEMM is the worst case of this class because it is a TVM COMPILED-IN FUNCTION rather than an installable package: BlockScaleQuantizeLinear.forward() probes extern.get_store().cutlass_gemm and tvm.get_global_func('cutlass.groupwise_scaled_gemm_e4m3fn_e4m3fn', allow_missing=True), and if either is absent the layer dequantizes and does a BF16 matmul. MLC-LLM's own source flags its arch coverage as unfinished: '# Todo: check "shape supported by cutlass" for Hopper'. ---- TWO ENGINES, SAME KERNEL FAMILY, DIFFERENT FLOORS ---- Marlin: vLLM's query_marlin_supported_quant_types returns [] when device_capability < 75; SGLang's identical-named function returns [] when device_capability < 80. On Turing (sm75) vLLM has Marlin and SGLang does not. The FP8-Marlin variants are looser than both: vLLM's is_fp8_marlin_supported() is just has_device_capability(75), and is_fp4_marlin_supported() is is_cuda() and has_device_capability(75). The same name means 'below 75 nothing' in one engine, 'below 80 nothing' in another, and '75 is enough' for the FP4 and FP8 variants. SGLang's MXFP4 MoE path adds a hard resource constraint on top: 'Current MXFP4 MoE layer is not supported by Marlin.' after check_moe_marlin_supports_layer(layer, 32, allow_tile_padding=True). ---- HARD-ERROR GATES, WHICH ARE THE GOOD ONES ---- Prefer these. SGLang raises rather than degrades on three counts, and every message names the way out: 'ModelOpt NVFP4 native dense GEMM backends require SM100+. Use --fp4-gemm-backend marlin on SM80-SM90.'; 'NVFP4 Marlin requires group_size=16, got %s.'; 'moe_runner_backend=flashinfer_mxfp4 requires SM90, SM100, or SM120.'; 'Unsupported kv_cache_dtype: %s.' vLLM raises with an aggregated reason string — choose_*_linear_kernel accumulates every kernel's failure reason and raises ValueError('Failed to find a kernel that can implement the ... Reasons: \n' + joined reasons) — which is the single most useful diagnostic in the stack, because it prints the whole rejected candidate list with a per-kernel reason. USE IT: when a load fails, read that string, it names the gate. TensorRT-LLM's is TLLM_CHECK_WITH_INFO(isMarlinNvfp4DenseSmSupported(sm), 'Marlin NVFP4 GEMM is only supported on SM89, SM90-99, and SM120/121; current SM = %d'). LMDeploy's published lists are a different failure mode again — not a gate at all but a stale transcription: its FP8 list names Ada and Hopper and does not mention Blackwell, and its INT8 list names H100 rather than H200. ---- THE RULE THIS RECORD EXISTS TO STATE ---- A support matrix answers 'can this format run on this GPU'. It cannot answer 'will it run natively here', because that is a conjunction of silicon AND runtime AND build AND optional package, and the four fail differently: silicon and runtime fail loudly or fall to a named reason; build and optional-package gates fail SILENTLY into a weight-only kernel that returns the right numbers. So before trusting a Y in any support table for FP8, MX or FP4, check four things and not one: the architecture is right; the CUDA version clears that format's per-arch floor (12.0 / 12.4 / 12.3 / 12.8 depending on format and whether the model is MoE); the wheel compiled the SM-specific macro; and the optional module for the fast path is actually installed. Four checks, and only the first is visible in the table you were given.