vLLM's 'Your GPU does not have native support for FP4' warning fires on weight-only NVFP4 checkpoints, and gets quoted as evidence the silicon lacks FP4 kernels
[minor] measurement
Fields
id
gotchas/nvfp4-marlin-warning-blames-gpu-for-weight-only-checkpointClass
measurement
Severity
minor
Concepts
cutlasscutlass-fp4marlin-moe
Symptom
prepare_nvfp4_moe_layer_for_marlin() emits 'Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.' on a Jetson AGX Thor (SM110) that demonstrably does have native FP4. Verified on that device with vLLM 0.27.1 / CUDA 13: torch.ops._C.cutlass_scaled_mm_supports_fp4(110) = True and cutlass_group_gemm_supported(110) = True. Serving a weight-only checkpoint (NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4) selects 'MARLIN' NvFp4 MoE backend and fires the warning; a true W4A4 checkpoint (nm-testing/nvfp4_moe-e2e) on the same device selects cutlass, no warning, and generates correctly.
Root cause
The Marlin NVFP4 MoE path is selected for two independent reasons - either the GPU genuinely lacks native FP4 cutlass support, OR the checkpoint is weight-only (W4A16_NVFP4 / MIXED_PRECISION, no FP4 activation scales), in which case a W4A4 cutlass kernel is inapplicable regardless of how capable the silicon is - and the warning only describes the first. The reporter notes the message is quoted as evidence that a GPU family lacks FP4 kernels and has been circulating as exactly that claim for SM110. An identical unconditional warning exists in compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w4a4_mxfp4.py.
Workaround
Do not read this warning as a hardware capability statement. Establish FP4 support directly with torch.ops._C.cutlass_scaled_mm_supports_fp4(<capability>) - that is the same predicate the NVFP4 linear path uses to choose its kernel. The fix (vLLM PR #53328) gates the warning on cutlass_fp4_supported() and states the weight-only case explicitly; before it lands, check the checkpoint's quant_algo field rather than trusting the log.
notes
A measurement gotcha rather than a serving failure: the model runs correctly in both cases. It matters because a wrong capability claim propagates - it is how SM110 came to be listed as lacking FP4 kernels - and because the ambiguity hides the real distinction (weight-only checkpoint vs W4A4 checkpoint), which is the thing that actually changes the kernel. Logging-only change, so the behavioral fix is cosmetic; the epistemic fix is not.
Notes
A measurement gotcha rather than a serving failure: the model runs correctly in both cases. It matters because a wrong capability claim propagates - it is how SM110 came to be listed as lacking FP4 kernels - and because the ambiguity hides the real distinction (weight-only checkpoint vs W4A4 checkpoint), which is the thing that actually changes the kernel. Logging-only change, so the behavioral fix is cosmetic; the epistemic fix is not.