vLLM per-format kernel capability matrix derived from source (which format has a NATIVE GEMM on which GPU generation, which silently degrades to emulation, and which is a verified negative)

GROUP SIZE LEFT NULL 2026-10-04 and NOT MEANINGFUL FOR THIS RECORD: the columns span NVFP4 group 16, MXFP4/MXFP6/MXFP8 group 32, and Marlin/Machete group 32/64/128/-1. Read the per-format records. PROXY ENUM VALUE - DECLARED 2026-10-04: a MATRIX record filed under 'nvfp4'; use the nvfp4 record for NVFP4 itself. REQUESTED ENUM ADDITION: a 'hardware-matrix' or 'support-matrix' enum value. WHAT THIS RECORD IS FOR: the two existing matrix records (qhw-trtllm-hardware-matrix and qhw-vllm-supported-hardware-matrix) are both TRANSCRIPTIONS OF VENDOR DOCS, and a transcription has no way to express the three things a capacity plan needs most: whether a cell is native or silently emulated, whether a CUDA/driver version gate applies on top of the architecture, and what the KV-cache rows do. This record fills the cells those two leave blank, derived from vLLM main-branch source rather than from any table. The single most important fact in it is that vLLM's kernel registries are ORDERED LISTS WITH A LAST-RESORT EMULATION ENTRY, so 'vLLM supports MXFP4 on my GPU' is very often true in the sense of 'it will load and run, by dequantizing to high precision and doing a BF16 matmul'. Any cell marked EMULATED below is that case. NATIVE GEMM, BY FORMAT AND GENERATION. NVFP4 (W4A4): native only on sm100/103 (B200, B300) and sm120/121 (RTX 5090, RTX PRO 6000). CutlassNvFp4LinearKernel gates on cutlass_scaled_mm_supports_fp4, whose C++ body returns false unless the CUDA runtime is >= 12.8 and then only for 100<=cc<120 or 120<=cc<130; anything else falls through to STD_TORCH_CHECK_NOT_IMPLEMENTED('No compiled nvfp4 mm kernel for SM <n>. Recompile with CUDA >= 12.8 and CC >= 100.'). H100/H200 (sm90), L40S/4090/6000-Ada (sm89/86), A100/A10 (sm80/86) and V100 (sm70) have NO native NVFP4 GEMM. On those parts vLLM serves an NVFP4 checkpoint through MarlinNvFp4LinearKernel (W4A16) or EmulationNvFp4LinearKernel - that is the already-recorded silent precision degradation, and this record puts a verified negative behind it for Ampere/Ada/Hopper rather than leaving it blank. MXFP4: on NVIDIA, native only where an OPTIONAL module is installed, and never by default. The CUDA candidate list is [FlashInferMxFp4LinearKernel, MarlinMxFp4LinearKernel, HummingMxFp4LinearKernel, B12xMxFp4LinearKernel, EmulationMxfp4LinearKernel]; FlashInferMxFp4LinearKernel.is_supported() is 'if current_platform.has_device_capability(100) and has_flashinfer_cutedsl()' - and has_device_capability is a >= comparison, so that gate PASSES ON sm120 as well as on sm100, meaning the only thing standing between an RTX 5090 and a native MXFP4 GEMM in vLLM main is whether the FlashInfer CuTe-DSL module happens to be installed. B12xMxFp4LinearKernel requires is_device_capability_family(120) plus the optional b12x package. MarlinMxFp4LinearKernel is weight-only and logs that it ignores the requested activation quantization, so selecting it is the W4A4-to-W4A16 degradation rather than a native path. HummingMxFp4LinearKernel requires the optional humming package at SM75+. So: sm100/103 and sm120/121 can reach a native MXFP4 GEMM via FlashInfer (given the module), while sm90 and below CANNOT reach one at all - Emulation is the only option there. The CUDA candidate list is [FlashInferMxFp4LinearKernel, MarlinMxFp4LinearKernel, HummingMxFp4LinearKernel, B12xMxFp4LinearKernel, EmulationMxfp4LinearKernel]; FlashInfer requires has_device_capability(100), B12x requires device_capability_family(120), and Emulation returns True unconditionally, with the engine's own warning text: 'The current platform does not support native MXFP4 computation. Using EmulationMxfp4LinearKernel, with simulated weight dequantization and activation QDQ (quantize and dequantize), with the linear layers computed in high precision.' THE PRACTICAL SHAPE: on sm120 and sm100 the outcome depends on optional installs, not on silicon; on sm90 and below emulation is unconditional. Note the sharp contrast with TensorRT-LLM, whose matrix marks MXFP4 an unconditional Y on sm120: same silicon, different engine, and in vLLM the Y is conditional on a Python package being present. This is exactly the case where reading one engine's table as the hardware's capability is wrong. MXFP6: HummingMxFp6LinearKernel (SM75+, optional package) and EmulationMxfp6LinearKernel, which accepts MXFP6 weights with MXFP4 or MXFP6 or unquantized activations. There is no CUTLASS or FlashInfer MXFP6 path in vLLM main, so MXFP6 is emulation wherever Humming is absent. MXFP8: native on sm100/103 via FlashInferCutlassMxfp8LinearKernel / FlashInferTrtllmMxfp8LinearKernel (both require has_device_capability(100)) or FlashInferCutedslMxfp8LinearKernel (requires device_capability_family(100)); on sm120 via B12xMxfp8LinearKernel (requires device_capability_family(120)) and FlashInfer's mm_mxfp8, which additionally requires N % 32 == 0 on SM12x; on sm80+ via MarlinMxfp8LinearKernel, whose gate is is_fp8_marlin_supported() = has_device_capability(75) and which is a WEIGHT-ONLY kernel (ModelOptMxFp8Config.get_min_capability() carries the comment 'Marlin kernel supports MXFP8 on SM80+'). Below sm75, or whenever the above are unavailable, EmulationMxfp8LinearKernel returns True and, by default, DEQUANTIZES AT LOAD TIME - the code says so: 'Dequantize MXFP8 -> BF16 ONCE here, at load time, so apply_weights runs a plain BF16 linear with no per-step dequant -- i.e. run as if from a BF16 checkpoint. The 1-byte MXFP8 weight is replaced by BF16 (2x its size...)' - so on that path you lose the memory saving too, not just the compute. MXFP8 is also the only MX format with a native AMD path: RocmDotScaledMxfp8LinearKernel requires supports_mx(), which on ROCm is gfx95 or gfx1250, described in the code as 'native MX requires CDNA4 (gfx95x)'; MI300-class gfx942 is NOT in that set and falls through to the BF16 emulation path. On the FP4/MX side AMD has AiterMxfp4LinearKernel behind supports_mx() plus a required quark package, and AITER FP8/INT8 kernels behind VLLM_ROCM_USE_AITER=1 and a compute-capability >= 90 style gate. NVFP4 on ROCm has ONLY EmulationNvFp4LinearKernel - there is no ROCm entry at all in _POSSIBLE_NVFP4_KERNELS. FP8, THREE SCALING MODES, THREE DIFFERENT SUPPORT SETS - this is the part the vendor docs flatten. FP8 per-tensor: the widest of the three, and CUTLASS-gated at has_device_capability(89) via cutlass_scaled_mm_supports_fp8, which returns false below SM89, demands CUDA >= 12.4 on SM89 and CUDA >= 12.0 on SM90. CudapPlatform.supports_fp8() is the same 89 floor. So Ada and Hopper are native, Ampere is not. IMPORTANT: Ampere is not merely unlisted here - it is handled by an explicitly named fallback: MarlinFP8ScaledMMLinearKernel, whose module comment reads 'For GPUs that lack FP8 hardware support, we can leverage the Marlin kernel for fast weight-only FP8 quantization', gated only at has_device_capability(75). An FP8 checkpoint on an A100 therefore LOADS AND RUNS, as weight-only FP8 through Marlin with FP16 activations - memory saving without the W8A8 compute saving. FP8 block-scaled (the 1x128/128x128 DeepSeek-style layout): cutlass_scaled_mm_supports_block_fp8 returns CUDA_VERSION>=12080 for cc>=100, CUDA_VERSION>=12000 for 90<=cc<100, and false for everything below SM90. So block-scaled FP8 is Hopper with CUDA 12.0+ and Blackwell sm100/103 with CUDA 12.8+, and NOT available on Ampere, Ada or sm120 through CUTLASS. This is the CUDA-version gate the vendor matrix cannot express. DeepGemmFp8BlockScaledMMKernel requires is_deep_gemm_supported(), whose failure reason is 'Currently, only Hopper and Blackwell GPUs are supported.' FP8 ROWWISE (per-token/per-channel dynamic): native per-token scaling via CUTLASS on sm90+ only (kFp8DynamicTokenSym is selected in Fp8LinearMethod only 'elif cutlass_fp8_supported()'), otherwise kFp8DynamicTensorSym - i.e. on a part where per-token is unavailable vLLM SILENTLY FALLS BACK TO PER-TENSOR ACTIVATION SCALING rather than failing. That is a real, quiet loss of accuracy headroom and it is invisible in any support table. The ROCm registry does contain RowWiseTorchFP8ScaledMMLinearKernel, and this repo already records the related RDNA4 rowwise-FP8 auto-selection trap in the gotcha vllm-rdna4-rowwise-fp8-kernel-auto-selected. W4A16 (AWQ and GPTQ): the one format family genuinely native from Turing up. MarlinLinearKernel.get_min_capability()=75 with group sizes [-1,32,64,128]; MacheteLinearKernel requires capability == 90 EXACTLY (not >=) with group sizes [-1,64,128] for FP16/BF16; ExllamaLinearKernel is float16-only; TritonW4A16LinearKernel.get_min_capability() returns 0 with the comment 'Triton handles capability checks itself' and group sizes [-1,32,64,128,256], making it the universal fallback on both CUDA and ROCm; ConchLinearKernel needs the optional conch-triton-kernels package and supports group sizes [-1,128] only. Turing and Ampere are covered by Marlin and Triton; Hopper by Machete. VOLTA IS THE VERIFIED NEGATIVE: query_marlin_supported_quant_types() returns an EMPTY LIST when device_capability < 75, and AutoAWQConfig.get_min_capability() returns 75, so AWQ and Marlin have no Volta path while GPTQ (get_min_capability()=60) does. That asymmetry is real and is the code-level reason 'GPTQ is the older format' is backwards as a compatibility claim. W4A8 (INT4 weights + INT8 activations) AND W4A8-FP8 (INT4 weights + FP8 e4m3 activations) ARE TWO DIFFERENT FORMATS IN vLLM AND ONLY ONE IS CPU-ONLY. W4A8-INT8: verified negative on every NVIDIA generation - the CUDA mixed-precision candidate list [CutlassW4A8, Machete, Marlin, Conch, Exllama, TritonW4A16, Humming] rejects the config in all seven cases and choose_mp_linear_kernel raises ValueError('Failed to find a kernel that can implement the WNA16 linear layer. Reasons: ...') at load time. It IS available on x86 CPU as well as Arm CPU, on two independent routes (Dynamic4bitLinearKernel with no x86 exclusion, and CPUWNA16LinearKernel's ops.int4_scaled_mm_cpu path gated on VLLM_CPU_INT4_W4A8 (default true) plus bf16 activations plus AMX tile or RISC-V) - see the sibling record qhw-vllm-supported-hardware-matrix for the full derivation. W4A8-FP8: the OPPOSITE shape - CompressedTensorsW4A8Fp8 is gated by _is_fp8_w4a8_sm90, which calls _check_scheme_supported(90, match_exact=True), so it requires capability == 90 EXACTLY. Hopper SM90 only; sm100/103/107 and sm120 are excluded by the equality check even though they are newer. Note that CutlassW4A8LinearKernel itself is also Hopper-exactly-90 and further requires act_type float8_e4m3fn, weight_type int4, zero_points False, group_size EXACTLY 128, K and N divisible by 128, and out_dtype bfloat16. KV CACHE - THE BLANK COLUMN IN BOTH EXISTING MATRIX RECORDS, NOW FILLED FROM CacheDType. The accepted --kv-cache-dtype values in vllm/config/cache.py are: auto, float16, bfloat16, fp8, fp8_e4m3, fp8_e5m2, fp8_inc, fp8_ds_mla, nvfp4_ds_mla, turboquant_k8v4, turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_3bit_nc, int4_per_token_head, int8_per_token_head, fp8_per_token_head, nvfp4, nvfp4_4over6. FP8 KV: CUDA 11.8+ for both e4m3 and e5m2, ROCm for e4m3 only, Gaudi via fp8_inc - stated in the CacheDtype field docstring verbatim. Per-head scaling ('fp8_per_token_head') requires the Flash Attention backend plus llm-compressor calibration, per the quantized_kvcache doc. NVFP4 KV: sm100/103 AND sm107 only, hard-gated - the FlashInfer backend raises ValueError('--kv-cache-dtype nvfp4 requires the SM100 trtllm-gen FlashInfer path.') unless supports_trtllm_attention() is true for both prefill and decode, and supports_trtllm_attention() returns is_device_capability_family(100) after special-casing sm90 and sm12x to 'not is_prefill'; since that family test is cc//10 == 10 it is true for 100..109 and therefore INCLUDES sm107. It also requires the NVIDIA artifactory to be reachable and is disabled under VLLM_BATCH_INVARIANT. Two structural limits in the same code: 'NVFP4 KV cache requires the trtllm-gen backend inside the wrapper; fa2/fa3 do not support nvfp4', and 'FlashInfer non-causal attention is not supported with NVFP4 KV cache'. There is also a bare 'assert not self.is_kvcache_nvfp4' on the attention-sinks prefill branch that covers sm90 and sm12x. CROSS-ENGINE DIFFERENCE WORTH FLAGGING: TensorRT-LLM's matrix marks NVFP4 KV Cache '.' for Rubin(sm107) while vLLM covers sm107 as supported; and TRT-LLM marks sm120 '.' too, which vLLM agrees with. 4-bit AND 3-bit KV ON NVIDIA: NO BUILT-IN PATH. The only sub-8-bit KV dtypes vLLM accepts on CUDA are the TurboQuant family (turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_3bit_nc, turboquant_k8v4, where NCC = no-compression residuals) and int4_per_token_head. There is no int4/3-bit KV dtype that is a plain block-quantized cache, and there is NO 2-bit KV dtype at all - which independently corroborates this repo's existing conclusion that nothing below 3-bit KV is offered by any engine. WHAT IS NATIVE ON AMD, BRIEFLY, BECAUSE THE TWO NVIDIA-NAMED MATRICES LEAVE AMD EMPTY AND 'EMPTY' READS LIKE 'UNKNOWN'. Native MXFP8 on gfx95/gfx1250 (CDNA4). Native MXFP4 on gfx95/gfx1250 behind AITER and a required quark package. Native FP8 on CDNA and RDNA4 via AITER, with a compute-capability >= 90 style gate on the AITER FP8 kernels. Native INT8 via AiterInt8ScaledMMLinearKernel and TritonInt8ScaledMMLinearKernel. W4A16 via RDNA3W4A16LinearKernel (requires on_gfx1100(), with the failure reason 'RDNA3 W4A16 kernel requires gfx1100'), RDNAHybridW4A16LinearKernel, TritonW4A16LinearKernel and ExllamaLinearKernel ('Exllama is only supported on CUDA and ROCm'). NOT native anywhere on AMD: Marlin (not in the ROCm list at all), Machete, CUTLASS, and NVFP4 (emulation only). On AMD, FP8 dtype is FNUZ rather than OCP E4M3 for gfx94-class parts (is_fp8_fnuz returns 'gfx94' in _GCN_ARCH, fp8_dtype returns torch.float8_e4m3fnuz), which is a numerical difference a capacity plan should know about before comparing an AMD FP8 number with a Blackwell one. THE EMULATION COLUMN, AND WHY IT IS THE DANGEROUS ONE. EmulationMxfp4LinearKernel, EmulationMxfp6LinearKernel, EmulationMxfp8LinearKernel and EmulationNvFp4LinearKernel ALL return is_supported() == (True, None) unconditionally, with the engine comment on the NVFP4 one: 'Always available as a last-resort fallback.' They are in the candidate list for every platform, they are last, and nothing warns the operator that compute has silently become high-precision - except a logger.warning_once that is easy to miss in a startup log. This is the same class of failure as the already-recorded Marlin-for-NVFP4 swap, generalised: on vLLM main, MXFP4 on consumer Blackwell sm120, MXFP6 without Humming, and MXFP8 on anything below sm75 are all 'supported' in the sense that the model loads. THE OPERATIONAL RULE THIS RECORD IMPLIES, and it is the one thing to carry away: for MX formats, 'vLLM runs it' is not a hardware statement. Check which kernel the startup log names.