inference/research
verifiedmedium · 0.80updated 2026-10-04

GGUF IQ1_M (1.75 bpw trellis-flavoured 1-bit with 3-bit sub-block scales and a grid-shift bit)

2.1460 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '1.75 bpw quantization'. block_iq1_m = { uint8_t qs[QK_K/8]; uint8_t qh[QK_K/16]; uint8_t scales[QK_K/32]; } = 32 + 16 + 8 = 56 bytes / 256 = 1.75 bpw

Fields

id
quantization/gguf-iq1-m
Scheme
gguf-iq1-m
Bits
2.1460 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '1.75 bpw quantization'. block_iq1_m = { uint8_t qs[QK_K/8]; uint8_t qh[QK_K/16]; uint8_t scales[QK_K/32]; } = 32 + 16 + 8 = 56 bytes / 256 = 1.75 bpw
Weight group size
32
Kernels
ggml dequantize_row_iq1_m + get_rowsggml-cuda dequantize_mul_mat_vec / convert_mul_mat
Quality delta
Official tool string '1.75 bpw quantization' (qlab2-quantize-options-table); no controlled fp16 delta published. Null + note.
notes
CORRECTED 2026-10-04: was 32 (correct) - now with a kernel basis. dequantize_row_iq1_m applies TWO 3-bit sub-scales per 32 values (`dl1` and `dl2` unpacked from one uint16 of sc[], for ib < QK_K/32). 32-value scale group, 8 sub-scales per 256 super-block. This is also the ONLY block struct in ggml-common.h with no fp16 at all - the super-block scale lives in the separate iq1m_scale_t union reconstructed from the top bits of scales[0..3], and the header notes the scale field is 3-bit only BECAUSE QK_K is 256 (it would be 4-bit at QK_K=64), so this type is not portable across a QK_K change. SCHEMA MISMATCH: there is no ggml/iq* value in the quantization scheme enum, so this record uses the closest existing value 'gguf-q4_k_m' (or 'gguf-q2_k') and puts the true type in `name` and `bits`. Requested enum additions are listed in the task report. Weight-only by construction: activations are promoted to q8_K (an intermediate dot-product format, not a storage format) and the matmul runs at fp16/bf16, so native_support is deliberately EMPTY and the whole accelerator ecosystem is emulated_support. No accelerator has an i-quant or k-quant path in silicon. COMMON TO ALL IQ-QUANTS, verified from ggml/src/ggml-common.h: they use the QK_K = 256 super-block like the K-quants, but instead of a UNIFORM integer grid they quantize to indices into a per-block CODEBOOK of 256 learned non-uniform 4-bit values (the 'i' in i-quant is for the codebook grid / i-quant from the ik_llama.cpp lineage). The index lookup is table-driven at dequant time, which is why they compress better than K-quants at the same nominal bits and why they dequantize more slowly - the dot product needs a gather from the codebook. This is a real throughput trade, not a free win. BACKEND SITUATION, verified: the CUDA library shipped in llamafile releases is size-optimized and OMITS the IQ-quant kernels (IQ1_*, IQ2_*, IQ3_*, IQ4_*); offloading an IQ-quantized model to an NVIDIA GPU therefore leaves those layers on the CPU with correct output but no GPU acceleration (qlab2-llamafile-support-docs). The Metal and ROCm builds are NOT size-optimized and include full IQ-quant GPU support - so an RX 7900 XTX on ROCm/HIP is in a materially better IQ position than an RTX 4090 on a stock llamafile build. Vulkan has no dedicated IQ-quant path. Upstream llama.cpp has per-type dequant kernels; ik_llama.cpp additionally ships the Trellis family (IQ1_KT/IQ2_KT/IQ3_KT/IQ4_KT, integer-base trellis, CUDA PR 113 / Metal PR 475 / NEON PR 471 / CPU PR 441) and _R4/_R16 row-interleaved variants. IQ1_M IS THE ODD ONE OUT IN THE BLOCK TABLE: it is the ONLY quant struct in ggml-common.h with NO ggml_half at all - no fp16 scale, not even a super-block one. The header comments describe the planes exactly: qs[QK_K/8] = 32 bytes is the 'grid index, low 8 bits'; qh[QK_K/16] = 16 bytes is the 'grid index, high 3 bits + grid shift bit (for two groups of 8)'; scales[QK_K/32] = 8 bytes are '3-bit block scales (4-bit if QK_K == 64)'. 32+16+8 = 56 bytes / 256 = 1.75 bpw. The scale quantization is deliberately coupled to the block size - the comment says the scale field is 3-bit only because QK_K is 256, and would be 4-bit at QK_K = 64 - so 'iq1_m' is not portable across a QK_K change, unlike the other types. The separate iq1m_scale_t union in the header exists to hold the per-group shift. For 8-expert models IQ1_M promotes to IQ2_XXS, same as IQ1_S. Separately, ik_llama.cpp adds Trellis quants IQ1_KT/IQ2_KT/IQ3_KT/IQ4_KT on a novel INTEGER-BASE trellis with CUDA (PR 113), Metal (PR 475), NEON (PR 471) and CPU (PR 441) implementations - that is a different lineage from upstream ggml and is recorded separately rather than conflated here. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'gguf-q2_k' (a proxy recorded while the enum lacked a legal value) and is now 'gguf-iq1-m'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: LLAMA.CPP INTEGER STORAGE TYPE, SO STRUCTURALLY ABSENT. Every GGUF type here is a packed integer block format read by a dequant kernel (dequantize_row_*) and then executed as FP16; ggml publishes no int4 or sub-8-bit tensor-core path that consumes these blocks directly. The llama.cpp CUDA and Vulkan backends both run these formats by dequantizing. Consequence for a reader: nothing further to verify - and note the contrast with nvfp4/mxfp4/mxfp8 in this repo, which ARE hardware formats on Blackwell and carry populated native_support lists, so '4-bit' alone does not predict this field. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

CORRECTED 2026-10-04: was 32 (correct) - now with a kernel basis. dequantize_row_iq1_m applies TWO 3-bit sub-scales per 32 values (`dl1` and `dl2` unpacked from one uint16 of sc[], for ib < QK_K/32). 32-value scale group, 8 sub-scales per 256 super-block. This is also the ONLY block struct in ggml-common.h with no fp16 at all - the super-block scale lives in the separate iq1m_scale_t union reconstructed from the top bits of scales[0..3], and the header notes the scale field is 3-bit only BECAUSE QK_K is 256 (it would be 4-bit at QK_K=64), so this type is not portable across a QK_K change. SCHEMA MISMATCH: there is no ggml/iq* value in the quantization scheme enum, so this record uses the closest existing value 'gguf-q4_k_m' (or 'gguf-q2_k') and puts the true type in `name` and `bits`. Requested enum additions are listed in the task report. Weight-only by construction: activations are promoted to q8_K (an intermediate dot-product format, not a storage format) and the matmul runs at fp16/bf16, so native_support is deliberately EMPTY and the whole accelerator ecosystem is emulated_support. No accelerator has an i-quant or k-quant path in silicon. COMMON TO ALL IQ-QUANTS, verified from ggml/src/ggml-common.h: they use the QK_K = 256 super-block like the K-quants, but instead of a UNIFORM integer grid they quantize to indices into a per-block CODEBOOK of 256 learned non-uniform 4-bit values (the 'i' in i-quant is for the codebook grid / i-quant from the ik_llama.cpp lineage). The index lookup is table-driven at dequant time, which is why they compress better than K-quants at the same nominal bits and why they dequantize more slowly - the dot product needs a gather from the codebook. This is a real throughput trade, not a free win. BACKEND SITUATION, verified: the CUDA library shipped in llamafile releases is size-optimized and OMITS the IQ-quant kernels (IQ1_*, IQ2_*, IQ3_*, IQ4_*); offloading an IQ-quantized model to an NVIDIA GPU therefore leaves those layers on the CPU with correct output but no GPU acceleration (qlab2-llamafile-support-docs). The Metal and ROCm builds are NOT size-optimized and include full IQ-quant GPU support - so an RX 7900 XTX on ROCm/HIP is in a materially better IQ position than an RTX 4090 on a stock llamafile build. Vulkan has no dedicated IQ-quant path. Upstream llama.cpp has per-type dequant kernels; ik_llama.cpp additionally ships the Trellis family (IQ1_KT/IQ2_KT/IQ3_KT/IQ4_KT, integer-base trellis, CUDA PR 113 / Metal PR 475 / NEON PR 471 / CPU PR 441) and _R4/_R16 row-interleaved variants. IQ1_M IS THE ODD ONE OUT IN THE BLOCK TABLE: it is the ONLY quant struct in ggml-common.h with NO ggml_half at all - no fp16 scale, not even a super-block one. The header comments describe the planes exactly: qs[QK_K/8] = 32 bytes is the 'grid index, low 8 bits'; qh[QK_K/16] = 16 bytes is the 'grid index, high 3 bits + grid shift bit (for two groups of 8)'; scales[QK_K/32] = 8 bytes are '3-bit block scales (4-bit if QK_K == 64)'. 32+16+8 = 56 bytes / 256 = 1.75 bpw. The scale quantization is deliberately coupled to the block size - the comment says the scale field is 3-bit only because QK_K is 256, and would be 4-bit at QK_K = 64 - so 'iq1_m' is not portable across a QK_K change, unlike the other types. The separate iq1m_scale_t union in the header exists to hold the per-group shift. For 8-expert models IQ1_M promotes to IQ2_XXS, same as IQ1_S. Separately, ik_llama.cpp adds Trellis quants IQ1_KT/IQ2_KT/IQ3_KT/IQ4_KT on a novel INTEGER-BASE trellis with CUDA (PR 113), Metal (PR 475), NEON (PR 471) and CPU (PR 441) implementations - that is a different lineage from upstream ggml and is recorded separately rather than conflated here. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'gguf-q2_k' (a proxy recorded while the enum lacked a legal value) and is now 'gguf-iq1-m'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: LLAMA.CPP INTEGER STORAGE TYPE, SO STRUCTURALLY ABSENT. Every GGUF type here is a packed integer block format read by a dequant kernel (dequantize_row_*) and then executed as FP16; ggml publishes no int4 or sub-8-bit tensor-core path that consumes these blocks directly. The llama.cpp CUDA and Vulkan backends both run these formats by dequantizing. Consequence for a reader: nothing further to verify - and note the contrast with nvfp4/mxfp4/mxfp8 in this repo, which ARE hardware formats on Blackwell and carry populated native_support lists, so '4-bit' alone does not predict this field. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references