inference/research
verifiedhigh · 0.95updated 2026-10-05

vLLM linear kernel registry ordering (_POSSIBLE_* dicts in vllm/model_executor/kernels/linear/__init__.py) as read 2026-10-05: Marlin precedes Emulation on the MX and NVFP4 paths

repo

Fields

id
sources/w6q-vllm-kernel-registry-ordering-2026-10
Publisher
vLLM project
Kind
repo
Accessed
2026-10-05
notes
Read 2026-10-05 from vLLM main @ 528772a4. THIS CORRECTS A WAVE-4 RECORD. Wave 4 (qhw-vllm-kernel-capability-matrix) described the CUDA MXFP4 list as [FlashInfer, Marlin, Humming, B12x, Emulation] and the NVFP4 fallback story as 'Marlin or Emulation'. Both are still true as ORDER, but wave 4's framing understated the consequence, and the registries have since grown. The current CUDA lists, quoted verbatim: _POSSIBLE_MXFP4_KERNELS = [FlashInferMxFp4LinearKernel, MarlinMxFp4LinearKernel, HummingMxFp4LinearKernel, B12xMxFp4LinearKernel, EmulationMxfp4LinearKernel]. _POSSIBLE_MXFP6_KERNELS = [HummingMxFp6LinearKernel, EmulationMxfp6LinearKernel]. _POSSIBLE_NVFP4_KERNELS = [FlashInferCuteDslNvFp4LinearKernel, FlashInferCutlassNvFp4LinearKernel, FlashInferB12xNvFp4LinearKernel, CutlassNvFp4LinearKernel, FlashInferCuteDslNvFp4W4A16LinearKernel, MarlinNvFp4LinearKernel, FlashInferTrtllmNvFp4LinearKernel, FlashInferCudnnNvFp4LinearKernel, FbgemmNvFp4LinearKernel, B12xNvFp4LinearKernel, TorchNvFp4LinearKernel, EmulationNvFp4LinearKernel, HummingNvFp4LinearKernel] — with the in-code comment 'Weight-only: a fallback on a W4A4-capable checkpoint, not a preference.' directly above the FlashInferCuteDslNvFp4W4A16 entry. _POSSIBLE_MXFP8_KERNELS = [FlashInferCutedsl, FlashInferCutlass, Marlin, B12x, Emulation, Humming, FlashInferTrtllm]. The load-bearing consequence, which no wave-4 record states: because Marlin sits at index 1 (MXFP4), index 1 (MXFP8) and index 5 (NVFP4), and Marlin's own gate is only is_fp4_marlin_supported()/is_fp8_marlin_supported() = is_cuda() and has_device_capability(75), the DEFAULT vLLM fallback for MXFP4, MXFP8 and NVFP4 on any SM75+ part is the WEIGHT-ONLY W4A16 Marlin kernel, not the Emulation kernel. Emulation is reached only when Marlin is also unavailable (sub-SM75, ROCm, or MXFP6 on ROCm) or is displaced by --linear-backend. So 'vLLM falls back to emulation' is the wrong default reading for MXFP4/MXFP8/NVFP4 on NVIDIA; the correct statement is 'vLLM falls back to Marlin W4A16, which keeps the memory saving and drops the W4A4 compute saving; emulation, which additionally gives up the memory saving for MXFP8 at load time, is the second-order fallback.' The MarlinMxFp4LinearKernel can_implement makes the swap explicit and non-silent in the log: it logs 'MarlinMxFp4LinearKernel is a weight-only (A16) kernel; the requested activation quantization (%s) is ignored.' Also newly present in this registry and not in any repo record: a --linear-backend map (_LINEAR_BACKEND_KERNEL_MAP) with named backends cutlass / marlin / triton / flashinfer_cutedsl / flashinfer_trtllm / flashinfer_cudnn / flashinfer_b12x / humming / deep_gemm / torch / aiter / machete / fbgemm / conch / exllama / emulation / xpu / xpu_woq, and the resolution policy _resolve_backend_kernels which, when the requested backend has no kernel for the layer type at hand, logs '--linear-backend=%s was requested, but no %s kernel exists for %s layers; falling back to normal kernel selection for this layer.' and proceeds with the unfiltered list rather than failing. That is an explicitly non-fatal misconfiguration path and belongs in the gate list.

Notes

Read 2026-10-05 from vLLM main @ 528772a4. THIS CORRECTS A WAVE-4 RECORD. Wave 4 (qhw-vllm-kernel-capability-matrix) described the CUDA MXFP4 list as [FlashInfer, Marlin, Humming, B12x, Emulation] and the NVFP4 fallback story as 'Marlin or Emulation'. Both are still true as ORDER, but wave 4's framing understated the consequence, and the registries have since grown. The current CUDA lists, quoted verbatim: _POSSIBLE_MXFP4_KERNELS = [FlashInferMxFp4LinearKernel, MarlinMxFp4LinearKernel, HummingMxFp4LinearKernel, B12xMxFp4LinearKernel, EmulationMxfp4LinearKernel]. _POSSIBLE_MXFP6_KERNELS = [HummingMxFp6LinearKernel, EmulationMxfp6LinearKernel]. _POSSIBLE_NVFP4_KERNELS = [FlashInferCuteDslNvFp4LinearKernel, FlashInferCutlassNvFp4LinearKernel, FlashInferB12xNvFp4LinearKernel, CutlassNvFp4LinearKernel, FlashInferCuteDslNvFp4W4A16LinearKernel, MarlinNvFp4LinearKernel, FlashInferTrtllmNvFp4LinearKernel, FlashInferCudnnNvFp4LinearKernel, FbgemmNvFp4LinearKernel, B12xNvFp4LinearKernel, TorchNvFp4LinearKernel, EmulationNvFp4LinearKernel, HummingNvFp4LinearKernel] — with the in-code comment 'Weight-only: a fallback on a W4A4-capable checkpoint, not a preference.' directly above the FlashInferCuteDslNvFp4W4A16 entry. _POSSIBLE_MXFP8_KERNELS = [FlashInferCutedsl, FlashInferCutlass, Marlin, B12x, Emulation, Humming, FlashInferTrtllm]. The load-bearing consequence, which no wave-4 record states: because Marlin sits at index 1 (MXFP4), index 1 (MXFP8) and index 5 (NVFP4), and Marlin's own gate is only is_fp4_marlin_supported()/is_fp8_marlin_supported() = is_cuda() and has_device_capability(75), the DEFAULT vLLM fallback for MXFP4, MXFP8 and NVFP4 on any SM75+ part is the WEIGHT-ONLY W4A16 Marlin kernel, not the Emulation kernel. Emulation is reached only when Marlin is also unavailable (sub-SM75, ROCm, or MXFP6 on ROCm) or is displaced by --linear-backend. So 'vLLM falls back to emulation' is the wrong default reading for MXFP4/MXFP8/NVFP4 on NVIDIA; the correct statement is 'vLLM falls back to Marlin W4A16, which keeps the memory saving and drops the W4A4 compute saving; emulation, which additionally gives up the memory saving for MXFP8 at load time, is the second-order fallback.' The MarlinMxFp4LinearKernel can_implement makes the swap explicit and non-silent in the log: it logs 'MarlinMxFp4LinearKernel is a weight-only (A16) kernel; the requested activation quantization (%s) is ignored.' Also newly present in this registry and not in any repo record: a --linear-backend map (_LINEAR_BACKEND_KERNEL_MAP) with named backends cutlass / marlin / triton / flashinfer_cutedsl / flashinfer_trtllm / flashinfer_cudnn / flashinfer_b12x / humming / deep_gemm / torch / aiter / machete / fbgemm / conch / exllama / emulation / xpu / xpu_woq, and the resolution policy _resolve_backend_kernels which, when the requested backend has no kernel for the layer type at hand, logs '--linear-backend=%s was requested, but no %s kernel exists for %s layers; falling back to normal kernel selection for this layer.' and proceeds with the unfiltered list rather than failing. That is an explicitly non-fatal misconfiguration path and belongs in the gate list.

Cross-references

This record references no other records.