SGLang quantization kernel selection source (fp4_utils.py, fp8_utils.py, marlin_utils.py, modelopt_quant.py, utils/common.py arch predicates) read from main @ v0.5.21

Read 2026-10-05 from sgl-project/sglang main (repo pushed 2026-10-05T15:17:33Z; latest release v0.5.21 published 2026-10-02T01:04Z). SGLang's quantization dispatch is NOT structured like vLLM's ordered kernel registries — it is a single enum with an if/elif ladder — and it carries gates vLLM does not. (1) THE CUDA-VERSION GATE NO SUPPORT TABLE EXPRESSES: python/sglang/srt/utils/common.py defines a family of lru_cache'd arch predicates, all built on _check_cuda_device_version(device_capability_majors=[...], cuda_version=(M,N)). is_blackwell_supported = is_blackwell = majors [10, 11, 12] AND cuda_version >= (12, 8). is_sm120_supported = majors [12] AND >= (12, 8). is_sm100_supported = majors [10] AND >= (12, 8). is_sm100_or_sm110_supported = majors [10, 11] AND >= (12, 8). is_sm90_supported = majors [9] AND >= (12, 3). is_sm80_supported = majors [8] AND >= (11, 0). is_ampere_with_cuda_12_3 = majors [8] AND >= (12, 3). is_hopper_with_cuda_12_3 = majors [9] AND >= (12, 3). Plus two exact-match predicates that the major-based helper cannot express, with the comment 'GB10 (DGX Spark and OEM equivalents). Not expressible with _check_cuda_device_version, which only matches on the major.': is_sm120() == capability (12,0) exactly and is_sm121() == (12,1) exactly. So SGLang's own 'is Blackwell' is a CUDA-12.8-AND-Blackwell test, not a silicon test. (2) FP4 GEMM BACKEND LADDER, quoted from initialize_fp4_gemm_config() in fp4_utils.py: `if backend == 'auto': if get_platform().is_sm100: backend = 'flashinfer_cutedsl'; elif is_cuda() and (10, 0) > get_device_capability() >= (8, 0): backend = 'marlin'; else: backend = 'flashinfer_cutlass'`. Note the range is EXCLUSIVE at 10.0 and INCLUSIVE at 8.0, so Marlin takes sm80-sm99 and sm100 does NOT fall into it; everything else — including sm70 and sm120 — lands in flashinfer_cutlass. This refines the wave-4 'marlin on SM80-SM90' summary: the window is sm80-sm99, and sm120 is NOT Marlin in SGLang. (3) MODELOPT NVFP4 HARD FAILURE rather than silent fallback, quoted from modelopt_quant.py process_weights_after_loading: `if not get_platform().is_blackwell: raise ValueError('ModelOpt NVFP4 native dense GEMM backends require SM100+. Use --fp4-gemm-backend marlin on SM80-SM90.')` — and it sits AFTER an early `if get_fp4_gemm_runner_backend().is_marlin(): ... return`, which additionally raises ValueError('NVFP4 Marlin requires group_size=16, got ...') and ('NVFP4A16 Marlin requires group_size=16, got ...'). So the Marlin NVFP4 route is group-size-locked to 16 and the native route requires is_blackwell (CUDA 12.8). (4) MARLIN'S FLOOR IS 80 IN SGLANG AND 75 IN vLLM — quoted from marlin_utils.py query_marlin_supported_quant_types: `if device_capability < 80: return []`. vLLM's equivalent (marlin_utils.py) is `if device_capability < 75: return []`. Same kernel family, two engines, a five-point disagreement about where Marlin starts; on sm75 (Turing) the two engines therefore disagree about whether Marlin exists. (5) SGLANG'S OWN CUTLASS FP8 PREDICATE, quoted from fp8_utils.py cutlass_fp8_supported(): `major, minor = get_device_capability(); cuda_version = get_cuda_version(); if major >= 9: return cuda_version >= (12, 0); elif major == 8 and minor == 9: return cuda_version >= (12, 4); return False`. Same numbers as vLLM's C++ but expressed as a Python predicate, and note the `major >= 9` branch means sm100/103/107/120 inherit the Hopper rule exactly as vLLM's does. (6) SGLANG'S FLASHINFER PER-TENSOR FP8 FLOOR IS A THREE-ARCH LIST, quoted: `return is_flashinfer_available() and (get_platform().is_sm90 or get_platform().is_sm100 or get_platform().is_sm120)`. (7) SGLANG MXFP4 MoE IS ARCH-EXCLUSIVE AND HARD-FAILS OFF THE LIST, quoted from mxfp4.py: SM100 -> 'trtllm_sm100', SM120 -> 'cutlass_sm120', SM90 -> 'cutlass_sm90' but only if _FI_HAS_SM90_CUTLASS_MXFP4 else RuntimeError('moe_runner_backend=flashinfer_mxfp4 on SM90 requires the interleave_moe_{weights,scales}_for_sm90_mixed_gemm helpers from FlashInfer PR #3084 (>= 0.6.11). Upgrade flashinfer-python or pick a different backend (e.g. marlin / triton_kernel).'), and everything else -> NotImplementedError('moe_runner_backend=flashinfer_mxfp4 requires SM90, SM100, or SM120.'). That is a FlashInfer *version* gate (>= 0.6.11) layered on the arch gate. (8) SGLANG KV-CACHE DTYPES ARE A MUCH SHORTER LIST THAN vLLM's — mem_cache/kv_cache_dtype.py configure_kv_cache_dtype accepts only auto, fp8_e5m2, fp8_e4m3, mxfp8, bf16/bfloat16, and nvfp4/fp4_mx_block16, with 'else: raise ValueError(f"Unsupported kv_cache_dtype: {server_args_kv_cache_dtype}.")'. There is NO int4_per_token_head, NO int8_per_token_head, NO fp8_per_token_head, NO turboquant_* and no fp8_ds_mla/nvfp4_ds_mla in this function. Independently corroborating the repo's existing conclusion that no engine offers a 2-bit KV dtype, SGLang's list has no sub-8-bit entry at all except nvfp4. Its nvfp4 branch carries its own version gate: it requires hasattr(torch, 'float4_e2m1fn_x2') else raise ValueError(f'--kv-cache-dtype={server_args_kv_cache_dtype} requires torch.float4_e2m1fn_x2 support. Please use PyTorch 2.8.0+ with CUDA 12.8+.'), and it emits a WARNING at request time: '%s KV Cache might lead to an accuracy drop!'. The fp4_kv_cache_quant_method.py module then narrows the *compute* side to sm100 only (get_platform().is_sm100 appears at three sites, including 'if get_platform().is_sm100 and k_scale != 1.0'), so on SGLang an NVFP4 KV cache is an SM100-only path.