inference/research
verifiedhigh · 0.95updated 2026-10-04

vLLM quantization kernel source (vllm/model_executor/kernels/linear/*) - per-format is_supported()/can_implement() capability gates

repo

Fields

id
sources/w4q-vllm-quant-kernel-source
Publisher
vLLM project
Kind
repo
Accessed
2026-10-04
notes
Read from the vLLM main branch 2026-10-04. The per-format kernel registries live in vllm/model_executor/kernels/linear/__init__.py as _POSSIBLE_KERNELS (mixed-precision WNA16/W4A8), _POSSIBLE_NVFP4_KERNELS, _POSSIBLE_MXFP4_KERNELS, _POSSIBLE_MXFP8_KERNELS, _POSSIBLE_MXFP6_KERNELS, _POSSIBLE_FP8_KERNELS, _POSSIBLE_FP8_BLOCK_KERNELS, _POSSIBLE_INT8_KERNELS, _POSSIBLE_WFP8A16_KERNELS. Each registry is a dict keyed by PlatformEnum (CUDA, ROCM, XPU, CPU) giving the ordered candidate kernel list. Selection is mechanical: choose_*_linear_kernel() iterates the platform list in order, skips kernels whose get_min_capability() exceeds the device compute capability, calls kernel.is_supported(cc) and kernel.can_implement(config), and returns the FIRST kernel that passes; if none pass it raises ValueError('Failed to find a kernel that can implement the ... Reasons: ...' + per-kernel reasons). This makes the capability gates below authoritative rather than advisory: a kernel that reports False never gets chosen, and a format with no passing kernel is a hard failure at model load, not a silent fallback. Gates captured verbatim: MarlinLinearKernel.get_min_capability()=75 and can_implement requires is_cuda(); MacheteLinearKernel requires is_device_capability(90) exactly ('Machete requires compute capability of 90 (Hopper)'); CutlassW4A8LinearKernel.get_min_capability()=90 and can_implement requires is_device_capability(90) exactly, act_type float8_e4m3fn, zero_points False, weight_type int4, group_size 128, K and N divisible by 128, out_dtype bfloat16; Marlin get_min_capability()=75, MARLIN_SUPPORTED_GROUP_SIZES=[-1,32,64,128], query_marlin_supported_quant_types() returns [] for device_capability<75 (which is the code-level source of the docs table's Volta NO for Marlin); ExllamaLinearKernel only float16 activations; TritonW4A16LinearKernel.get_min_capability()=0 ('Triton handles capability checks itself'), group sizes [-1,32,64,128,256], so it is the universal W4A16 fallback; CPUWNA16LinearKernel.get_min_capability()=-1 and can_implement requires is_cpu(); Dynamic4bitLinearKernel.get_min_capability()=1, can_implement requires is_cpu(), SUPPORTED_QUANT_TYPES=[scalar_types.int4] (asymmetric int4), and on Arm additionally requires float32/bfloat16/float16 activations and torch.ops.aten._dyn_quant_matmul_4bit (KleidiAI); MarlinNvFp4LinearKernel/MarlinMxfp4LinearKernel gate on is_fp4_marlin_supported() = is_cuda() and has_device_capability(75); MarlinFP8ScaledMMLinearKernel/MarlinMxfp8LinearKernel gate on is_fp8_marlin_supported() = has_device_capability(75); CutlassNvFp4LinearKernel gates on cutlass_fp4_supported() -> ops.cutlass_scaled_mm_supports_fp4; TorchNvFp4LinearKernel requires is_device_capability_family(100) plus torch._scaled_mm and torch.float4_e2m1fn_x2; FlashInferCutlassMxfp8/FlashInferTrtllmMxfp8LinearKernel require has_device_capability(100); FlashInferCutedslMxfp8LinearKernel requires is_device_capability_family(100); DeepGemmMxfp8BmmLinearKernel requires is_device_capability_family(100); B12xNvFp4/B12xMxFp4/B12xMxfp8/B12xTensorFP8/B12xFp8BlockScaled kernels all require is_device_capability_family(120); Humming* kernels (MXFP4/MXFP6/MXFP8/NVFP4/FP8/INT8/WNA16) require is_cuda() and has_device_capability(75); Emulation* kernels return True unconditionally and are therefore the terminal fallback for MXFP4, MXFP6, MXFP8 and NVFP4 on every platform; AiterMxfp4LinearKernel requires supports_mx() which on ROCm is gfx95 or gfx1250 only; RocmDotScaledMxfp8LinearKernel requires supports_mx() ('native MX requires CDNA4 (gfx95x)'); XPUMxFp4/XPUMxFp8/XPUW4A8Int kernels are XPU-only.

Notes

Read from the vLLM main branch 2026-10-04. The per-format kernel registries live in vllm/model_executor/kernels/linear/__init__.py as _POSSIBLE_KERNELS (mixed-precision WNA16/W4A8), _POSSIBLE_NVFP4_KERNELS, _POSSIBLE_MXFP4_KERNELS, _POSSIBLE_MXFP8_KERNELS, _POSSIBLE_MXFP6_KERNELS, _POSSIBLE_FP8_KERNELS, _POSSIBLE_FP8_BLOCK_KERNELS, _POSSIBLE_INT8_KERNELS, _POSSIBLE_WFP8A16_KERNELS. Each registry is a dict keyed by PlatformEnum (CUDA, ROCM, XPU, CPU) giving the ordered candidate kernel list. Selection is mechanical: choose_*_linear_kernel() iterates the platform list in order, skips kernels whose get_min_capability() exceeds the device compute capability, calls kernel.is_supported(cc) and kernel.can_implement(config), and returns the FIRST kernel that passes; if none pass it raises ValueError('Failed to find a kernel that can implement the ... Reasons: ...' + per-kernel reasons). This makes the capability gates below authoritative rather than advisory: a kernel that reports False never gets chosen, and a format with no passing kernel is a hard failure at model load, not a silent fallback. Gates captured verbatim: MarlinLinearKernel.get_min_capability()=75 and can_implement requires is_cuda(); MacheteLinearKernel requires is_device_capability(90) exactly ('Machete requires compute capability of 90 (Hopper)'); CutlassW4A8LinearKernel.get_min_capability()=90 and can_implement requires is_device_capability(90) exactly, act_type float8_e4m3fn, zero_points False, weight_type int4, group_size 128, K and N divisible by 128, out_dtype bfloat16; Marlin get_min_capability()=75, MARLIN_SUPPORTED_GROUP_SIZES=[-1,32,64,128], query_marlin_supported_quant_types() returns [] for device_capability<75 (which is the code-level source of the docs table's Volta NO for Marlin); ExllamaLinearKernel only float16 activations; TritonW4A16LinearKernel.get_min_capability()=0 ('Triton handles capability checks itself'), group sizes [-1,32,64,128,256], so it is the universal W4A16 fallback; CPUWNA16LinearKernel.get_min_capability()=-1 and can_implement requires is_cpu(); Dynamic4bitLinearKernel.get_min_capability()=1, can_implement requires is_cpu(), SUPPORTED_QUANT_TYPES=[scalar_types.int4] (asymmetric int4), and on Arm additionally requires float32/bfloat16/float16 activations and torch.ops.aten._dyn_quant_matmul_4bit (KleidiAI); MarlinNvFp4LinearKernel/MarlinMxfp4LinearKernel gate on is_fp4_marlin_supported() = is_cuda() and has_device_capability(75); MarlinFP8ScaledMMLinearKernel/MarlinMxfp8LinearKernel gate on is_fp8_marlin_supported() = has_device_capability(75); CutlassNvFp4LinearKernel gates on cutlass_fp4_supported() -> ops.cutlass_scaled_mm_supports_fp4; TorchNvFp4LinearKernel requires is_device_capability_family(100) plus torch._scaled_mm and torch.float4_e2m1fn_x2; FlashInferCutlassMxfp8/FlashInferTrtllmMxfp8LinearKernel require has_device_capability(100); FlashInferCutedslMxfp8LinearKernel requires is_device_capability_family(100); DeepGemmMxfp8BmmLinearKernel requires is_device_capability_family(100); B12xNvFp4/B12xMxFp4/B12xMxfp8/B12xTensorFP8/B12xFp8BlockScaled kernels all require is_device_capability_family(120); Humming* kernels (MXFP4/MXFP6/MXFP8/NVFP4/FP8/INT8/WNA16) require is_cuda() and has_device_capability(75); Emulation* kernels return True unconditionally and are therefore the terminal fallback for MXFP4, MXFP6, MXFP8 and NVFP4 on every platform; AiterMxfp4LinearKernel requires supports_mx() which on ROCm is gfx95 or gfx1250 only; RocmDotScaledMxfp8LinearKernel requires supports_mx() ('native MX requires CDNA4 (gfx95x)'); XPUMxFp4/XPUMxFp8/XPUW4A8Int kernels are XPU-only.

Cross-references

This record references no other records.