llama.cpp MMQ quantized GEMM kernel source (mmq.cuh)
repo
Fields
id
sources/kern-llamacpp-mmq-sourcePublisher
ggml-org
Kind
repo
Accessed
2026-10-03
notes
Kernel source header (MIT). Confirms llama.cpp's quantized GEMM path quantizes activations to q8_1 blocks of 32 and accumulates in INT32, with optional DP4A (MMQ_DP4A_MAX_BATCH_SIZE 64, i.e. DP4A only while fp16 tensor cores exist), per-arch tuning headers (mmq-config-ampere, -blackwell, -cdna, -gcn, -rdna2/3/4/5, -pascal-dp4a), MMQ_ITER_K 256 (512 for FP4), MMQ_NWARPS 8, three q8_1 scale/data layouts, and a dedicated block_fp4_mmq type. Practical consequence: llama.cpp reaches quantized speed through integer SIMT dot products, not through FP8/NVFP4 tensor cores, so its quantized decode tok/s is not comparable to a Marlin/Machete number.
Notes
Kernel source header (MIT). Confirms llama.cpp's quantized GEMM path quantizes activations to q8_1 blocks of 32 and accumulates in INT32, with optional DP4A (MMQ_DP4A_MAX_BATCH_SIZE 64, i.e. DP4A only while fp16 tensor cores exist), per-arch tuning headers (mmq-config-ampere, -blackwell, -cdna, -gcn, -rdna2/3/4/5, -pascal-dp4a), MMQ_ITER_K 256 (512 for FP4), MMQ_NWARPS 8, three q8_1 scale/data layouts, and a dedicated block_fp4_mmq type. Practical consequence: llama.cpp reaches quantized speed through integer SIMT dot products, not through FP8/NVFP4 tensor cores, so its quantized decode tok/s is not comparable to a Marlin/Machete number.
Cross-references
This record references no other records.