inference/research
drafthigh · 0.90updated 2026-10-03

llama.cpp MMQ quantized GEMM kernel source (mmq.cuh)

repo
This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
sources/kern-llamacpp-mmq-source
Publisher
ggml-org
Kind
repo
Accessed
2026-10-03
notes
Kernel source header (MIT). Confirms llama.cpp's quantized GEMM path quantizes activations to q8_1 blocks of 32 and accumulates in INT32, with optional DP4A (MMQ_DP4A_MAX_BATCH_SIZE 64, i.e. DP4A only while fp16 tensor cores exist), per-arch tuning headers (mmq-config-ampere, -blackwell, -cdna, -gcn, -rdna2/3/4/5, -pascal-dp4a), MMQ_ITER_K 256 (512 for FP4), MMQ_NWARPS 8, three q8_1 scale/data layouts, and a dedicated block_fp4_mmq type. Practical consequence: llama.cpp reaches quantized speed through integer SIMT dot products, not through FP8/NVFP4 tensor cores, so its quantized decode tok/s is not comparable to a Marlin/Machete number.

Notes

Kernel source header (MIT). Confirms llama.cpp's quantized GEMM path quantizes activations to q8_1 blocks of 32 and accumulates in INT32, with optional DP4A (MMQ_DP4A_MAX_BATCH_SIZE 64, i.e. DP4A only while fp16 tensor cores exist), per-arch tuning headers (mmq-config-ampere, -blackwell, -cdna, -gcn, -rdna2/3/4/5, -pascal-dp4a), MMQ_ITER_K 256 (512 for FP4), MMQ_NWARPS 8, three q8_1 scale/data layouts, and a dedicated block_fp4_mmq type. Practical consequence: llama.cpp reaches quantized speed through integer SIMT dot products, not through FP8/NVFP4 tensor cores, so its quantized decode tok/s is not comparable to a Marlin/Machete number.

Cross-references

This record references no other records.