GGUF IQ3_XXS (3.0625 bpw, almost-true 3-bit with a 16-bit scale per 256)
Fields
quantization/gguf-iq3-xxsNotes
CORRECTED 2026-10-04: same basis as iq2_xxs. block_iq3_xxs is `{ ggml_half d; uint8_t qs[3*QK_K/8]; }` with a single fp16 d over the whole 256-value super-block and NO sub-block scale plane (the scales_and_signs pointer in the kernel aliases into the tail of qs, not a separate scales array). Header comment confirms: 3.0625 bpw 'because of the 16-bit scale for each block of 256'. SCHEMA MISMATCH: there is no ggml/iq* value in the quantization scheme enum, so this record uses the closest existing value 'gguf-q4_k_m' (or 'gguf-q2_k') and puts the true type in `name` and `bits`. Requested enum additions are listed in the task report. Weight-only by construction: activations are promoted to q8_K (an intermediate dot-product format, not a storage format) and the matmul runs at fp16/bf16, so native_support is deliberately EMPTY and the whole accelerator ecosystem is emulated_support. No accelerator has an i-quant or k-quant path in silicon. COMMON TO ALL IQ-QUANTS, verified from ggml/src/ggml-common.h: they use the QK_K = 256 super-block like the K-quants, but instead of a UNIFORM integer grid they quantize to indices into a per-block CODEBOOK of 256 learned non-uniform 4-bit values (the 'i' in i-quant is for the codebook grid / i-quant from the ik_llama.cpp lineage). The index lookup is table-driven at dequant time, which is why they compress better than K-quants at the same nominal bits and why they dequantize more slowly - the dot product needs a gather from the codebook. This is a real throughput trade, not a free win. BACKEND SITUATION, verified: the CUDA library shipped in llamafile releases is size-optimized and OMITS the IQ-quant kernels (IQ1_*, IQ2_*, IQ3_*, IQ4_*); offloading an IQ-quantized model to an NVIDIA GPU therefore leaves those layers on the CPU with correct output but no GPU acceleration (qlab2-llamafile-support-docs). The Metal and ROCm builds are NOT size-optimized and include full IQ-quant GPU support - so an RX 7900 XTX on ROCm/HIP is in a materially better IQ position than an RTX 4090 on a stock llamafile build. Vulkan has no dedicated IQ-quant path. Upstream llama.cpp has per-type dequant kernels; ik_llama.cpp additionally ships the Trellis family (IQ1_KT/IQ2_KT/IQ3_KT/IQ4_KT, integer-base trellis, CUDA PR 113 / Metal PR 475 / NEON PR 471 / CPU PR 441) and _R4/_R16 row-interleaved variants. IQ3_XXS mirrors IQ2_XXS's design at 3 bits: the header comment is '(Almost) "true" 3-bit quantization. Due to the need to use blocks as per ggml design, it ends up using 3.0625 bpw because of the 16-bit scale for each block of 256.' Structurally qs[3*QK_K/8] = 96 bytes of packed 3-bit codebook indices plus one fp16 d - no sub-block scale plane. MIXTURE: llama-quant.cpp promotes attn_k and attn_q to IQ2_S for this ftype (a genuine 2-bit attention, the only place an i-quant ftype drops attention below its base), attn_v to IQ3_S only when an imatrix is absent and n_gqa < 4 (otherwise Q4_K), ffn_down to Q4_K/Q3_K in the first n_layer/8 layers only when no imatrix is present, and IQ3_S on attn_output. The !has_imatrix conditions appear four separate times in this one branch, which is the clearest single piece of evidence in the codebase that imatrix changes the type schedule and not only the rounding. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'gguf-q3_k_m' (a proxy recorded while the enum lacked a legal value) and is now 'gguf-iq3-xxs'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: LLAMA.CPP INTEGER STORAGE TYPE, SO STRUCTURALLY ABSENT. Every GGUF type here is a packed integer block format read by a dequant kernel (dequantize_row_*) and then executed as FP16; ggml publishes no int4 or sub-8-bit tensor-core path that consumes these blocks directly. The llama.cpp CUDA and Vulkan backends both run these formats by dequantizing. Consequence for a reader: nothing further to verify - and note the contrast with nvfp4/mxfp4/mxfp8 in this repo, which ARE hardware formats on Blackwell and carry populated native_support lists, so '4-bit' alone does not predict this field. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.