inference/research
verifiedhigh · 0.90updated 2026-10-04

Marlin INT4 weight-only kernel (FP16xINT4 mixed-precision GEMM)

w4a16 (4-bit weights, FP16 activations)

Fields

id
quantization/qhw-marlin-kernel
Scheme
gptq
Bits
w4a16 (4-bit weights, FP16 activations)
Weight group size
128
Activation scheme
FP16 activations, no quantization. Weights are INT4 with per-group FP16 scales (group size 128). The kernel dequantizes weights to FP16 in registers before the tensor-core multiply, so the compute is FP16 x FP16, not INT4 x INT4.
Kernels
Marlin CUDA kernel (marlin/marlin_cuda_kernel.cu)vLLM marlin format (awq_marlin, gptq_marlin)SGLang marlin backend
Quality delta
From the Marlin README benchmark table (Llama2 models, INT4 group=128): 7B WikiText-2 5.12 (FP16) vs 5.27 (INT4), RedPajama 6.14 vs 6.30, MMLU 41.80 vs 40.07. 13B: WikiText-2 4.57 vs 4.67, RedPajama 5.67 vs 5.79, MMLU 52.10 vs 51.13. 70B: WikiText-2 3.12 vs 3.21, RedPajama 4.74 vs 4.81, MMLU 65.43 vs 64.81. These are GPTQ-quantized models evaluated with Marlin kernels, so the delta includes both quantization error and any kernel-specific effects.
notes
PROXY ENUM VALUE - DECLARED 2026-10-04: Marlin is a KERNEL, not a format. It is filed under 'gptq' because Marlin consumes GPTQ-packed weights and the enum has no kernel-family values. The proxy is a near miss on target (Marlin genuinely serves GPTQ and AWQ checkpoints, never KV cache or activations), but a query for 'which formats use gptq' will return Marlin, and a reader may conclude Marlin is a quantization FORMAT rather than a GEMM kernel that consumes one. The load-bearing fact is in this record's own notes and is invisible to a query: despite being called an 'INT4 kernel', the actual tensor-core operation is FP16 x FP16 - weights are dequantized to FP16 in registers before the multiply. REQUESTED ENUM ADDITION: a 'marlin' value in the kernel family. Marlin is the canonical INT4 weight-only kernel for Ampere/Ada GPUs. KEY PRECISION FACT: despite being called an 'INT4 kernel', the actual tensor-core operation is FP16 x FP16 - weights are dequantized to FP16 in registers before the multiply. This is NOT an INT4 tensor-core path (which does not exist on Ampere/Ada). The speedup comes from reduced memory bandwidth (4-bit weight storage), not from faster arithmetic. PERFORMANCE: the README claims 'near-ideal 4x speedups up to batchsizes of 16-32 tokens' on A10, with the 3.87x theoretical maximum (accounting for 0.125 bits/weight scale storage overhead). The kernel uses striped partitioning, L2 cache optimization, and double buffering to sustain this across batch sizes. CRITICAL CAVEAT: Marlin is NOT optimized for Hopper (SM90) - the README explicitly says 'Marlin is not yet optimized for Hopper'. On Hopper, Machete is the preferred kernel. The vLLM compatibility matrix shows Marlin supports Turing (SM7.5) and above, but NOT Volta (SM7.0). SGLang uses Marlin as the W4A16 fallback for NVFP4 checkpoints on SM80-SM90. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

PROXY ENUM VALUE - DECLARED 2026-10-04: Marlin is a KERNEL, not a format. It is filed under 'gptq' because Marlin consumes GPTQ-packed weights and the enum has no kernel-family values. The proxy is a near miss on target (Marlin genuinely serves GPTQ and AWQ checkpoints, never KV cache or activations), but a query for 'which formats use gptq' will return Marlin, and a reader may conclude Marlin is a quantization FORMAT rather than a GEMM kernel that consumes one. The load-bearing fact is in this record's own notes and is invisible to a query: despite being called an 'INT4 kernel', the actual tensor-core operation is FP16 x FP16 - weights are dequantized to FP16 in registers before the multiply. REQUESTED ENUM ADDITION: a 'marlin' value in the kernel family. Marlin is the canonical INT4 weight-only kernel for Ampere/Ada GPUs. KEY PRECISION FACT: despite being called an 'INT4 kernel', the actual tensor-core operation is FP16 x FP16 - weights are dequantized to FP16 in registers before the multiply. This is NOT an INT4 tensor-core path (which does not exist on Ampere/Ada). The speedup comes from reduced memory bandwidth (4-bit weight storage), not from faster arithmetic. PERFORMANCE: the README claims 'near-ideal 4x speedups up to batchsizes of 16-32 tokens' on A10, with the 3.87x theoretical maximum (accounting for 0.125 bits/weight scale storage overhead). The kernel uses striped partitioning, L2 cache optimization, and double buffering to sustain this across batch sizes. CRITICAL CAVEAT: Marlin is NOT optimized for Hopper (SM90) - the README explicitly says 'Marlin is not yet optimized for Hopper'. On Hopper, Machete is the preferred kernel. The vLLM compatibility matrix shows Marlin supports Turing (SM7.5) and above, but NOT Volta (SM7.0). SGLang uses Marlin as the W4A16 fallback for NVFP4 checkpoints on SM80-SM90. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (4)