inference/research
verifiedmedium · 0.78updated 2026-10-04

GGUF Q4_0 (legacy block-4 symmetric)

4.5 (classic block format; Q4_0 measured 4.4597 bpw in the IQ sweep, Q4_0 as a legacy 4-bit type is quoted as ~4)

Fields

id
quantization/gguf-q4-0
Scheme
gguf-q4_0
Bits
4.5 (classic block format; Q4_0 measured 4.4597 bpw in the IQ sweep, Q4_0 as a legacy 4-bit type is quoted as ~4)
Weight group size
32
Kernels
ggml dequantize_row_q4_0 + get_rowsggml-cuda mul_mat_vec_q4_0_q8_0 / mul_mat_q4_0_q8_1ggml-cpu AVX2/AVX-512 dequant+dot kernels
Quality delta
Llama-3.1-8B-Instruct (quant-gguf-kquant-study): WikiText-2 ppl 7.74 vs 7.32 F16; GSM8K 71.03 vs 77.63 (-6.60); IFEval 71.88 vs 78.93; mean 67.98 vs 69.47; size 4438 MiB (-71.03%).
notes
CONFIRMED 2026-10-04 against the kernel: QK4_0 = 32, one fp16 d per 32-value block, no zero point. Legacy format: symmetric block-wise 4-bit over blocks of K=32, one FP16 scale per block, no zero-point, codes in [-8,7] packed as nibbles (quant-gguf-kquant-study). It predates k-quants and has no super-block hierarchy. Q4_0 is *worse* on perplexity than both Q4_K_S (7.62) and Q4_K_M (7.56) at comparable size - the direct evidence that format design beats the nominal bit-width label. Weight-only: activations stay in FP16/BF16 at runtime, so there is no '4-bit activations with hardware support' claim to make here. ggml has bespoke dequantize+gemm kernels per type, but no accelerator has a GGUF/k-quant path in silicon; everything is dequantized on the fly. vLLM explicitly labels its GGUF support experimental and under-optimized and positions it as a memory-footprint tool (quant-vllm-gguf-doc). Q4_0 does have good prompt-processing throughput (97.35 pp512 vs 79.57 F16 in that study) but poor decode (4.36 vs 2.83). LEGACY BLOCK LAYOUT, verified from ggml/src/ggml-common.h: QK4_0 = 32 values per block, block_q4_0 = { ggml_half d; uint8_t qs[QK4_0/2]; } = 2 + 16 = 18 bytes / 32 values = 4.5 bpw, symmetric (delta only, no zero point). QK8_0 = 32, block_q8_0 = { ggml_half d; int8_t qs[QK8_0]; } = 2 + 32 = 34 bytes / 32 = 8.5 bpw. Both are 32-value blocks with a single fp16 scale - that is the entire difference from the K-quants, which use 256-value super-blocks with quantized sub-block scales. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: LLAMA.CPP INTEGER STORAGE TYPE, SO STRUCTURALLY ABSENT. Every GGUF type here is a packed integer block format read by a dequant kernel (dequantize_row_*) and then executed as FP16; ggml publishes no int4 or sub-8-bit tensor-core path that consumes these blocks directly. The llama.cpp CUDA and Vulkan backends both run these formats by dequantizing. Consequence for a reader: nothing further to verify - and note the contrast with nvfp4/mxfp4/mxfp8 in this repo, which ARE hardware formats on Blackwell and carry populated native_support lists, so '4-bit' alone does not predict this field. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

CONFIRMED 2026-10-04 against the kernel: QK4_0 = 32, one fp16 d per 32-value block, no zero point. Legacy format: symmetric block-wise 4-bit over blocks of K=32, one FP16 scale per block, no zero-point, codes in [-8,7] packed as nibbles (quant-gguf-kquant-study). It predates k-quants and has no super-block hierarchy. Q4_0 is *worse* on perplexity than both Q4_K_S (7.62) and Q4_K_M (7.56) at comparable size - the direct evidence that format design beats the nominal bit-width label. Weight-only: activations stay in FP16/BF16 at runtime, so there is no '4-bit activations with hardware support' claim to make here. ggml has bespoke dequantize+gemm kernels per type, but no accelerator has a GGUF/k-quant path in silicon; everything is dequantized on the fly. vLLM explicitly labels its GGUF support experimental and under-optimized and positions it as a memory-footprint tool (quant-vllm-gguf-doc). Q4_0 does have good prompt-processing throughput (97.35 pp512 vs 79.57 F16 in that study) but poor decode (4.36 vs 2.83). LEGACY BLOCK LAYOUT, verified from ggml/src/ggml-common.h: QK4_0 = 32 values per block, block_q4_0 = { ggml_half d; uint8_t qs[QK4_0/2]; } = 2 + 16 = 18 bytes / 32 values = 4.5 bpw, symmetric (delta only, no zero point). QK8_0 = 32, block_q8_0 = { ggml_half d; int8_t qs[QK8_0]; } = 2 + 32 = 34 bytes / 32 = 8.5 bpw. Both are 32-value blocks with a single fp16 scale - that is the entire difference from the K-quants, which use 256-value super-blocks with quantized sub-block scales. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: LLAMA.CPP INTEGER STORAGE TYPE, SO STRUCTURALLY ABSENT. Every GGUF type here is a packed integer block format read by a dequant kernel (dequantize_row_*) and then executed as FP16; ggml publishes no int4 or sub-8-bit tensor-core path that consumes these blocks directly. The llama.cpp CUDA and Vulkan backends both run these formats by dequantizing. Consequence for a reader: nothing further to verify - and note the contrast with nvfp4/mxfp4/mxfp8 in this repo, which ARE hardware formats on Blackwell and carry populated native_support lists, so '4-bit' alone does not predict this field. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references