vLLM Implementation x Hardware support matrix (which quantization implementation runs on which architecture, and where it does NOT)

GROUP SIZE LEFT NULL 2026-10-04 and NOT MEANINGFUL FOR THIS RECORD: it spans nine implementation rows covering per-tensor, per-channel, 64, 128 and 32 group conventions. Read the per-format records for group sizes. PROXY ENUM VALUE - DECLARED 2026-10-04: this is a MATRIX record spanning nine implementation rows (AWQ, GPTQ, Marlin, llm-compressor INT8 W8A8 and W4A8, llm-compressor FP8 W8A8, bitsandbytes, DeepSpeedFP, GGUF), filed under 'gptq'. GPTQ is the right anchor for a reason worth stating: it is the only row in the matrix that covers Volta, so it is the format whose availability actually varies most across the architectures listed. But a query for GPTQ returns this matrix rather than the format record; use the gptq record for GPTQ itself. REQUESTED ENUM ADDITION: a 'hardware-matrix' or 'support-matrix' value distinct from any format.THE COMPLEMENT TO THE TensorRT-LLM MATRIX, AND IT CORRECTS THE COMMONEST MISREADING. Columns Volta, Turing, Ampere, Ada, Hopper, AMD GPU, Intel GPU, x86 CPU, Arm CPU. Rows verbatim, YES/NO: AWQ = no Volta, yes Turing, yes Ampere, yes Ada, yes Hopper, NO AMD GPU, yes Intel GPU, yes x86 CPU, NO Arm CPU. GPTQ = yes Volta, yes Turing, yes Ampere, yes Ada, yes Hopper, NO AMD GPU, yes Intel GPU, yes x86 CPU, NO Arm CPU. Marlin (GPTQ/AWQ/FP8/FP4) = no Volta, Turing YES-WITH-A-FOOTNOTE, yes Ampere, yes Ada, yes Hopper, NO AMD GPU, NO Intel GPU, NO x86 CPU, NO Arm CPU. llm-compressor INT8 (W8A8) = no Volta, yes Turing, yes Ampere, yes Ada, yes Hopper, no AMD, no Intel, YES x86 CPU, no Arm. llm-compressor INT8 (W4A8) = NO on Volta, Turing, Ampere, Ada AND Hopper, NO AMD, NO Intel, and per the docs table NO x86 CPU, YES Arm CPU. llm-compressor FP8 (W8A8) = no Volta, no Turing, no Ampere, YES Ada, YES Hopper, no AMD, no Intel, no x86 CPU, no Arm. bitsandbytes = yes Volta through Hopper, no AMD/Intel/CPU. DeepSpeedFP = yes Volta through Hopper, no AMD/Intel/CPU. GGUF = yes Volta through Hopper, no AMD, no Intel, no CPU. === VERIFIED CELL BY CELL AGAINST vLLM'S OWN SOURCE 2026-10-04. Every cell below was re-derived from the code, because the code is what runs. The vendor table supplied the cells; the kernel source supplied the mechanism and, in one case, a correction. === *** ONE CELL FOUND WRONG: THE x86 CPU CELL OF THE W4A8 ROW. The docs table marks llm-compressor INT8 (W4A8) as NO on x86 CPU. The code says otherwise, on two independent routes, and this record's own previous reading ('W4A8 is Arm-CPU-only, so run it on the CPU is false on x86') inherited that docs error. *** Route 1, no ISA gate at all. CompressedTensorsW4A8Int.get_min_capability() returns 1 - the lowest value the kernel selector admits - and its create_weights() delegates to choose_mp_linear_kernel(). The CPU candidate list is [Dynamic4bitLinearKernel, ZentorchWNA16LinearKernel, CPUWNA16LinearKernel]. Dynamic4bitLinearKernel's own module docstring calls it 'the KleidiAI-accelerated w4a8int quantization scheme', and its can_implement() requires only is_cpu() and weight_type in [scalar_types.int4]; the float32/bfloat16/float16 activation restriction and the torch.ops.aten._dyn_quant_matmul_4bit availability probe both live INSIDE a 'if current_platform.get_cpu_architecture() == CpuArchEnum.ARM' branch, so they are Arm-only requirements and the kernel is not excluded on x86. Route 2, explicitly x86. CPUWNA16LinearKernel.process_weights_after_loading() sets layer.use_w4a8 = envs.VLLM_CPU_INT4_W4A8 (default True, per envs.py) and act_type == torch.bfloat16 and (torch.cpu._is_amx_tile_supported() or architecture is RISCV); when that holds it repacks to the AWQ blocked format via ops.convert_weight_packed_scale_zp(..., ops.CPUQuantAlgo.AWQ) and dispatches apply_weights to ops.int4_scaled_mm_cpu INSTEAD of ops.cpu_gemm_wna16. The x86 C++ op exists at csrc/cpu/sgl-kernels/gemm_int4.cpp: at::Tensor int4_scaled_mm_cpu(x, w, w_zeros, w_scales, bias) delegating to int4_scaled_mm_cpu_with_quant, which quantizes the activations itself ('act quant', 'activation quants into uint8_t', constexpr bool sym_quant_act = false) and fails with 'DA8W4: weight and input shapes mismatch' or 'int4_scaled_mm_cpu_with_quant: expect A to be bfloat16 or half.'. Its arch guards are CPU_CAPABILITY_AVX512 and CPU_CAPABILITY_RVV - x86 AVX512, not AMX, not Arm. CORRECTED CELL: llm-compressor INT8 (W4A8) = NO on Volta, Turing, Ampere, Ada and Hopper (confirmed correct by source), YES on x86 CPU with the caveat below, YES on Arm CPU. THE CAVEAT IS THE STORY, NOT A FOOTNOTE: the Arm route is the one the W4A8 recipe document advertises ('On Arm CPUs, this is accelerated through KleidiAI'), while on x86 it is reachable but conditional - the AMX/RVV route needs bf16 activations, and otherwise it is the AVX512 fallback. So the correct shape of this finding is the INVERSE of what this record previously said: 'run W4A8 on the CPU' is FALSE ON EVERY GPU AND TRUE ON x86, not false on x86. WHY THE FIVE NVIDIA COLUMNS ARE RIGHT - confirmed rather than assumed, and the failure mode is loud. The CUDA candidate list for a W4A8 config is [CutlassW4A8LinearKernel, MacheteLinearKernel, MarlinLinearKernel, ConchLinearKernel, ExllamaLinearKernel, TritonW4A16LinearKernel, HummingLinearKernel] and every one returns can_implement()=False: CutlassW4A8LinearKernel requires act_type == torch.float8_e4m3fn so an INT8-activation checkpoint fails it, MacheteLinearKernel requires is_device_capability(90) exactly plus FP16/BF16/FP8 activations, Marlin and Conch and Triton and Humming are weight-only (no input_quant_key), ExllamaLinearKernel 'only supports float16 activations'. The selection loop then raises ValueError('Failed to find a kernel that can implement the WNA16 linear layer. Reasons: ...') with a per-kernel reason list. A W4A8 checkpoint therefore cannot quietly run on a slower path on a GPU - it refuses to load. THE NEAR-MISS WORTH KNOWING: a W4A8 checkpoint whose ACTIVATIONS are FP8 e4m3 rather than INT8 is matched on Hopper by CompressedTensorsW4A8Fp8, gated by _is_fp8_w4a8_sm90 which calls _check_scheme_supported(90, match_exact=True) - capability must EQUAL 90, not merely exceed it. So 'W4A8' splits into an INT8-activation flavour that is CPU-only in vLLM and an FP8-activation flavour (W4A8-FP8) that is Hopper-exactly-90, which also excludes sm100/103/107 and sm120. FOUR OTHER ROWS RE-DERIVED FROM SOURCE, ALL CONFIRMING THE DOCS. (1) GPTQ IS THE ONLY ROW THAT COVERS VOLTA, and the code gives the mechanism: query_marlin_supported_quant_types() returns [] when device_capability < 75, which is exactly why AWQ's and Marlin's Volta cells are NO. AutoAWQConfig.get_min_capability() returns 75; AutoGPTQConfig.get_min_capability() returns 60 - the only quant config in the set with a sub-70 floor. (2) THE TURING FOOTNOTE IS NARROWER THAN THE CELL SUGGESTS: 'Turing does not support Marlin MXFP4'. MarlinLinearKernel.get_min_capability()=75 admits Turing for the INT4 path, and MarlinFP8 / MarlinNvFp4 / MarlinMxfp4 all gate on has_device_capability(75), so Marlin-minus-MXFP4 is the honest reading. (3) FP8 W8A8 IS ADA-AND-HOPPER-ONLY, confirmed four ways: CompressedTensorsW8A8Fp8.get_min_capability() returns 89 with the comment '# lovelace and up'; CudapPlatform.supports_fp8() returns has_device_capability(89); cutlass_scaled_mm_supports_fp8() returns false below SM89 and additionally demands CUDA 12.4 on SM89 versus CUDA 12.0 on SM90; and CompressedTensorsW8A8Int8.get_min_capability() returns 75 with '# turing and up' - so at Ampere the INT8 W8A8 row is YES while the FP8 W8A8 row is NO, at the same compute capability, which is the finding, not an artifact. (4) THE GROUP SIZES ARE REAL CONSTRAINTS, NOT COSMETIC: MARLIN_SUPPORTED_GROUP_SIZES = [-1, 32, 64, 128], Machete's are [-1, 64, 128] for FP16/BF16 and [-1, 128] otherwise, and Machete additionally requires in_features % 64 == 0 and out_features % 128 == 0 (MACHETE_PREPACKED_BLOCK_SHAPE = [64, 128]). THE AMD COLUMN IS A WHOLE-COLUMN BLANK THAT READS LIKE A DATA GAP. Every NVIDIA format row is NO on AMD GPU, which is TRUE for these particular vLLM BACKENDS but must not be read as 'AMD GPUs cannot do quantized inference'. The code shows why the column is empty for the NVIDIA-named rows: ROCm's mixed-precision candidate list is [RDNA3W4A16LinearKernel, RDNAHybridW4A16LinearKernel, TritonW4A16LinearKernel, ConchLinearKernel, ExllamaLinearKernel] - RDNA-targeted, no Marlin, no Machete, no CUTLASS. AMD's own path is Quark plus its FP8/INT8 kernels, covered by the separate quark and rocm-quant records in this repo. Related known gap already recorded in this repo: all Intel Gaudi quantization support has been migrated to vLLM-Gaudi, which is why Gaudi appears nowhere in this table. SCOPE AND THE CELLS THAT ARE STILL UNCHECKED, STATED AS SUCH. The table covers Volta through Hopper only - Blackwell is absent from it, which is why it must be read together with the TensorRT-LLM matrix in the sibling record rather than instead of it. Its own scope line warns the chart 'is subject to change as vLLM continues to evolve'. Five things remain genuinely unchecked rather than verified-negative, and are listed here so that nobody reads a null as a finding: (a) the Turing cell for AWQ and GPTQ specifically - get_min_capability()=75 for AWQ admits Turing, but which kernel is actually selected at SM75 was not traced; (b) whether bitsandbytes and DeepSpeedFP have any CPU or ROCm path in vLLM main - the table says no and nothing either way was found in the kernel registries, which is an absence of evidence, not a confirmed negative; (c) Intel GPU and Gaudi cells, untouched since Gaudi moved to vLLM-Gaudi; (d) every KV-cache cell, which this table does not cover at all - the code-level KV gates are in the sibling record qhw-vllm-kernel-capability-matrix; (e) the XPU column, which exists in the code (XPUMxFp4, XPUMxFp8, XPUW4A8IntLinearKernel) and has no row in this table at all.