inference/research
verifiedhigh · 0.90majorupdated 2026-10-04

A block format's declared block size is not its scale group, and a library default is not a format constant - reading either wrong silently corrupts the memory arithmetic

[major] format

Fields

id
gotchas/qzgw-block-size-is-not-the-super-block-and-a-default-is-not-a-constant
Class
format
Severity
major
Concepts
group sizescale sharingsuper-blockblocksizememory arithmeticquantizationproxy enumformat vs kernelkv-cache-quantkv-evictionkv-sparsity
Symptom
Two wrong-but-plausible weight_group_size values, both of which change every memory and quality number computed from them. (1) SEVEN GGUF records carried 256, which is the K-quant SUPER-BLOCK (QK_K) rather than the group over which one scale is shared: the true scale groups are 16 for q2_K, q3_K and q6_K, and 32 for q4_K, q5_K, iq4_xs and iq4_nl. (2) A library default read as a format constant: bitsandbytes NF4 carries 64, but the same NF4 data may be legally written at 32, 64, 128, 256, 512, 1024, 2048 or 4096, so describing NF4 as 64-group is false as a property of the format.
Root cause
The struct declaration and the kernel disagree about what size means, and the struct is the one people read. block_q4_K declares scales[K_SCALE_SIZE] = 12 bytes, but the question of how many weights share one scale is answered only by dequantize_row_q4_K, which does `for (j = 0; j < QK_K; j += 64)` consuming TWO get_scale_min_k4 scale/min pairs of 32 values each. The array length is a storage budget; the loop bound is the group. || The two-level scale design of the k-quants makes the trap worse than a simple off-by-one: a per-32 sub-block scale that is itself 6-bit and sits on an fp16 super-block scale is not a number a reader will guess, and 256 is simultaneously true (as the super-block) and useless (as the group). || blocksize in bitsandbytes is a caller-supplied OPTIONAL argument (group_size: Optional[int] = None, defaulting to 64) with eight legal values, so recording it as weight_group_size records a default. MLX has the same shape but worse: four modes at four different group sizes (affine 64, mxfp4 32, nvfp4 16, mxfp8 32), so a benchmark recorded as 4-bit on Apple hardware identifies neither a bit width nor a group. || The kernel set is narrower than the config space, which is the fact that makes the default look like a constant: csrc/kernels.cu instantiates kQuantizeBlockwise for 64 through 4096, but kQuantizeBlockwiseSmall - the 4-bit fast path - ONLY for qblock_size 32 and 64. The default sits inside the fast path, so almost every checkpoint anyone actually observes is at 64 unless they deliberately paid to go elsewhere.
Workaround
Read the scale group from the dequantize_row_* KERNEL, never from the block struct and never from the format name. The loop bounds are the answer: a bound of ib < QK_K/32 means 32; j += 64 with two scale fetches means 32; an index of l/16 means 16; and ib32 += 2 means 64. || For GGUF specifically, note that the k-quants are NOT 256-group while several i-quants with no sub-block scale plane (iq2_xxs, iq3_xxs, iq1_s) genuinely ARE 256 - because with one fp16 per super-block the super-block size and the scale group coincide. Check whether a sub-block scale plane exists before assuming either. || Treat null as a legitimate answer for this field. Per-tensor formats (fp8-e4m3, fp8-e5m2, int8 per-tensor) have no group BY DEFINITION; per-channel formats have the maximum possible group; KV, eviction and sparsity schemes have no weights at all; rotation and lattice-codebook methods (QuaRot, SpinQuant, QuIP#, AQLM) have no shared-scale group; and removed GGUF types have no surviving layout to read. Filling any of these with 128 or 256 manufactures a number that changes downstream memory arithmetic while looking entirely reasonable. || When a record must state null, state the reason in notes. The 53 nulls in this record set split into roughly three honest categories - not-applicable-by-construction, not-published, and not-a-format - and a reader cannot tell them apart from the value alone.
notes
The broader lesson is the one this repository has already been burned by twice, in a third disguise. The first burn was quantization records sitting on PROXY ENUM VALUES (nine KV-cache methods filed under weight formats), fixed by reclassifying 45 records. The second was a per-record flag with a correct value but an inverted boolean. This is the same failure shape again: a FIELD THAT IS CORRECT AND INVERTED BY COMPARISON. 256 is a correct number about these formats - it is simply the wrong quantity for this field - and it survives review because anyone checking whether 256 is a real ggml constant says yes. A field that stores a real number about the wrong axis is more dangerous than a field left blank, because it cannot be detected by looking for absence. THE PROXY-ENUM AUDIT THAT ACCOMPANIED THIS FOUND A SECOND, DISTINCT PROXY CLASS that the 45-record reclassification did not touch. The historical bug was a record filed under the wrong TARGET (a KV method under a weight format). The remaining class is a record filed under the wrong KIND: KERNELS, TOOLKITS and ALGORITHMS sitting under FORMAT enums. Marlin under gptq, Machete under awq, SpQR under gptq, SignRound under gptq, AutoRound under int8, Intel Neural Compressor under int8, ONNX Runtime GenAI model-builder under int8, Olive under int8. None of these misrepresents what the method TARGETS - they are all weight formats, never KV or activation, so the original failure (a query for which formats quantize the KV cache returning nonsense) cannot recur from them. But a query for which formats use GPTQ returns three records that are one kernel and two methods, and Marlin-under-gptq versus Machete-under-awq actively implies two different precisions when both produce identical FP16xFP16 W4A16 output. ONE RECORD IS ACTIVELY FALSE RATHER THAN MERELY APPROXIMATE: gguf-deprecated-q4-2-q4-3 is filed under gguf-q5-1. It documents two types REMOVED from ggml (type ids 4 and 5, stubs with .type_size = 0 and .is_quantized = false), so anyone selecting gguf-q5-1 from that filter is sent to a record stating that q5_1 is unrelated. Enum values requested rather than added, since shared schema files are parent-owned: exl3, mlx-affine, marlin, machete, spqr, signround, autoround, neural-compressor, olive, ortgenai-model-builder, and a slot for removed GGUF types. METHOD NOTE FOR THE NEXT AUDIT: the audit that produced this was mechanical rather than interpretive. For each record, check whether the value in weight_group_size is justified by a scale-shared-over-N loop bound in that format's own kernel, and check whether the record's notes contain any explanation for its enum choice. Eleven records had a scheme value their notes never justified. Both halves are scriptable, and the enum-justification half is the one that catches proxies: a record whose notes explain a proxy is fine, and a record whose notes do not is the finding.

Notes

The broader lesson is the one this repository has already been burned by twice, in a third disguise. The first burn was quantization records sitting on PROXY ENUM VALUES (nine KV-cache methods filed under weight formats), fixed by reclassifying 45 records. The second was a per-record flag with a correct value but an inverted boolean. This is the same failure shape again: a FIELD THAT IS CORRECT AND INVERTED BY COMPARISON. 256 is a correct number about these formats - it is simply the wrong quantity for this field - and it survives review because anyone checking whether 256 is a real ggml constant says yes. A field that stores a real number about the wrong axis is more dangerous than a field left blank, because it cannot be detected by looking for absence.

THE PROXY-ENUM AUDIT THAT ACCOMPANIED THIS FOUND A SECOND, DISTINCT PROXY CLASS that the 45-record reclassification did not touch. The historical bug was a record filed under the wrong TARGET (a KV method under a weight format). The remaining class is a record filed under the wrong KIND: KERNELS, TOOLKITS and ALGORITHMS sitting under FORMAT enums. Marlin under gptq, Machete under awq, SpQR under gptq, SignRound under gptq, AutoRound under int8, Intel Neural Compressor under int8, ONNX Runtime GenAI model-builder under int8, Olive under int8. None of these misrepresents what the method TARGETS - they are all weight formats, never KV or activation, so the original failure (a query for which formats quantize the KV cache returning nonsense) cannot recur from them. But a query for which formats use GPTQ returns three records that are one kernel and two methods, and Marlin-under-gptq versus Machete-under-awq actively implies two different precisions when both produce identical FP16xFP16 W4A16 output.

ONE RECORD IS ACTIVELY FALSE RATHER THAN MERELY APPROXIMATE: gguf-deprecated-q4-2-q4-3 is filed under gguf-q5-1. It documents two types REMOVED from ggml (type ids 4 and 5, stubs with .type_size = 0 and .is_quantized = false), so anyone selecting gguf-q5-1 from that filter is sent to a record stating that q5_1 is unrelated. Enum values requested rather than added, since shared schema files are parent-owned: exl3, mlx-affine, marlin, machete, spqr, signround, autoround, neural-compressor, olive, ortgenai-model-builder, and a slot for removed GGUF types.

METHOD NOTE FOR THE NEXT AUDIT: the audit that produced this was mechanical rather than interpretive. For each record, check whether the value in weight_group_size is justified by a scale-shared-over-N loop bound in that format's own kernel, and check whether the record's notes contain any explanation for its enum choice. Eleven records had a scheme value their notes never justified. Both halves are scriptable, and the enum-justification half is the one that catches proxies: a record whose notes explain a proxy is fine, and a record whose notes do not is the finding.

Cross-references