inference/research
verifiedmedium · 0.80updated 2026-10-04

GGUF Q4_1 / Q5_1 (legacy asymmetric 32-value blocks with per-block scale AND min)

4.5 bpw for q4_1 (QK4_1 = 32: 2 x fp16 + 16 = 36 bytes / 32 values), 5.5625 bpw for q5_1 (2 x fp16 + 4 + 16 = 40 bytes / 32)

Fields

id
quantization/gguf-legacy-q4-1
Scheme
gguf-q4-1
Bits
4.5 bpw for q4_1 (QK4_1 = 32: 2 x fp16 + 16 = 36 bytes / 32 values), 5.5625 bpw for q5_1 (2 x fp16 + 4 + 16 = 40 bytes / 32)
Weight group size
32
Kernels
ggml dequantize_row_q4_1 / dequantize_row_q5_1 + get_rowsggml-cuda dequantize_mul_mat_vec / convert_mul_mat
Quality delta
Official quantizer strings: Q4_0 '4.34G, +0.4685 ppl @ Llama-3-8B', Q4_1 '4.78G, +0.4511 ppl', Q5_0 '5.21G, +0.1316 ppl', Q5_1 '5.65G, +0.1062 ppl', Q8_0 '7.96G, +0.0026 ppl' (qlab2-quantize-options-table). All are the tool's own Llama-3-8B deltas; note Q4_1's +0.4511 vs Q4_0's +0.4685 is a 0.017 ppl gain for 0.44 GB.
notes
CONFIRMED 2026-10-04 against the kernel: QK4_1 = 32 and QK5_1 = 32. Unlike the symmetric q4_0/q5_0, q4_1/q5_1 are ASYMMETRIC - block_q4_1 carries BOTH a `d` and an `m` (min) fp16 per 32 values, so two fp16 per 32 = 5.5 bpw, and dequantize_row_q4_1 computes `d*(q&0xF) - m`. One scale pair per 32 values, no sub-block plane. Weight-only: activations are promoted to q8_K for the dot product and the matmul runs at fp16/bf16, so native_support is deliberately EMPTY and the whole accelerator ecosystem is emulated_support. No accelerator has a GGUF quant path in silicon - this is the single most commonly botched field in this repo. LAYOUT, verified from ggml/src/ggml-common.h: QK4_1 = 32 and block_q4_1 = { GGML_EXTENSION union { struct { ggml_half d; ggml_half m; } ; ggml_half2 dm; } ; uint8_t qs[QK4_1/2]; } = 2*sizeof(ggml_half) + 16 = 36 bytes / 32 values = 4.5 bpw. QK5_1 = 32 and block_q5_1 = same union + uint8_t qh[4] (the 5th bit) + uint8_t qs[QK5_1/2] = 2*2 + 4 + 16 = 40 bytes / 32 = 5.5625 bpw. The critical difference from q4_0/q5_0 is the MIN: x = d*q + m (asymmetric, zero-point-like) rather than x = d*q. This is the same affine form the k-quants use, at 32-value granularity with an fp16 min. STILL IN CIRCULATION and still selectable in the official quantizer: the tool lists Q4_1 at '4.78G, +0.4511 ppl @ Llama-3-8B' and Q5_1 at '5.65G, +0.1062 ppl' (qlab2-quantize-options-table) - both barely better than q4_0's '+0.4685 ppl' despite being wider, which is the practical reason nobody uses them. The one place llama.cpp still assigns them automatically is the imatrix guard in src/llama-quant.cpp: for Q4_0/Q5_0 WITH an imatrix, the first n_layer/8 ffn_down layers are bumped to Q4_1/Q5_1, with the in-source comment 'Guard against craziness in the first few ffn_down layers that can happen even with imatrix for Q4_0/Q5_0 ... Q4_1/Q5_1 do go crazy on ffn_down without an imatrix.' That comment is a direct, quotable statement that the affine min is what saves those tensors - and that it is unstable without calibration. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'gguf-q4_0' (a proxy recorded while the enum lacked a legal value) and is now 'gguf-q4-1'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

CONFIRMED 2026-10-04 against the kernel: QK4_1 = 32 and QK5_1 = 32. Unlike the symmetric q4_0/q5_0, q4_1/q5_1 are ASYMMETRIC - block_q4_1 carries BOTH a `d` and an `m` (min) fp16 per 32 values, so two fp16 per 32 = 5.5 bpw, and dequantize_row_q4_1 computes `d*(q&0xF) - m`. One scale pair per 32 values, no sub-block plane. Weight-only: activations are promoted to q8_K for the dot product and the matmul runs at fp16/bf16, so native_support is deliberately EMPTY and the whole accelerator ecosystem is emulated_support. No accelerator has a GGUF quant path in silicon - this is the single most commonly botched field in this repo. LAYOUT, verified from ggml/src/ggml-common.h: QK4_1 = 32 and block_q4_1 = { GGML_EXTENSION union { struct { ggml_half d; ggml_half m; } ; ggml_half2 dm; } ; uint8_t qs[QK4_1/2]; } = 2*sizeof(ggml_half) + 16 = 36 bytes / 32 values = 4.5 bpw. QK5_1 = 32 and block_q5_1 = same union + uint8_t qh[4] (the 5th bit) + uint8_t qs[QK5_1/2] = 2*2 + 4 + 16 = 40 bytes / 32 = 5.5625 bpw. The critical difference from q4_0/q5_0 is the MIN: x = d*q + m (asymmetric, zero-point-like) rather than x = d*q. This is the same affine form the k-quants use, at 32-value granularity with an fp16 min. STILL IN CIRCULATION and still selectable in the official quantizer: the tool lists Q4_1 at '4.78G, +0.4511 ppl @ Llama-3-8B' and Q5_1 at '5.65G, +0.1062 ppl' (qlab2-quantize-options-table) - both barely better than q4_0's '+0.4685 ppl' despite being wider, which is the practical reason nobody uses them. The one place llama.cpp still assigns them automatically is the imatrix guard in src/llama-quant.cpp: for Q4_0/Q5_0 WITH an imatrix, the first n_layer/8 ffn_down layers are bumped to Q4_1/Q5_1, with the in-source comment 'Guard against craziness in the first few ffn_down layers that can happen even with imatrix for Q4_0/Q5_0 ... Q4_1/Q5_1 do go crazy on ffn_down without an imatrix.' That comment is a direct, quotable statement that the affine min is what saves those tensors - and that it is unstable without calibration. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'gguf-q4_0' (a proxy recorded while the enum lacked a legal value) and is now 'gguf-q4-1'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references