llama.cpp MMQ: INT8 activation quantization with per-arch tile configs, no tensor cores
memory bound
Fields
id
flops/llamacpp-mmq-int8-dp4a-gemmClass
quantization_overhead
Bound by
memory
Arithmetic intensity
A different answer to the same problem, and the reason llama.cpp's quantized numbers are not comparable to Marlin's or Machete's. MMQ quantizes the ACTIVATIONS to q8_1 blocks of 32 elements and performs the dot product in INT8 accumulating to INT32, using DP4A where available. So the multiply-accumulates are integer SIMT instructions, not tensor-core MMAs - the FLOP:byte ratio the tensor cores exploit is not available at all. It buys portability: per-arch config headers exist for Ampere, Blackwell, CDNA, GCN, RDNA2/3/4/5 and Pascal (DP4A), the block size is a fixed 32 with three different scale/data layouts, MMQ_ITER_K is 256 (512 for FP4) and MMQ_NWARPS is 8. Notably, DP4A is used only while fp16 tensor cores are unavailable or the batch is small (MMQ_DP4A_MAX_BATCH_SIZE 64), which is itself a tensor-core-adjacent threshold.
Scales with
batch_sizeweight quantization type (a dedicated block_fp4_mmq type exists)num_SMs (MMQ_NWARPS 8 per block is a fixed occupancy choice)arch (per-arch tile configs differ)
Workarounds
Quantize activations to q8_1 blocks of 32 so the dequant cost is a shift per blockThree scale/data layout variants (D4, DS4, D2S6) chosen per quantization type to make the layout copyable to shared memory as a contiguous blockPer-arch tile configuration headers instead of one universal kernelMMQ_ITER_K 256, raised to 512 for FP4
notes
Practical decision input: on a consumer or datacenter GPU, llama.cpp reaches 4-bit throughput through integer SIMT dot products with a broad arch reach (including RDNA and CDNA), while Marlin and Machete reach it through tensor cores on a narrower arch list. Neither is strictly better - MMQ works on more parts, Machete reaches much higher tensor-core utilization where it applies - and any comparison must state which path was taken. Verified from the kernel header only; the existing llama.cpp engine and quantization records in this repo are the place for its benchmark numbers.
Notes
Practical decision input: on a consumer or datacenter GPU, llama.cpp reaches 4-bit throughput through integer SIMT dot products with a broad arch reach (including RDNA and CDNA), while Marlin and Machete reach it through tensor cores on a narrower arch list. Neither is strictly better - MMQ works on more parts, Machete reaches much higher tensor-core utilization where it applies - and any comparison must state which path was taken. Verified from the kernel header only; the existing llama.cpp engine and quantization records in this repo are the place for its benchmark numbers.