inference/research
verifiedhigh · 0.95minorupdated 2026-10-11

Quantization bit-width and format family names cited as format records

[minor] format

Fields

id
gotchas/gap-quantization-family-format-identifiers
Class
format
Severity
minor
Concepts
quantization-familyformat-gap2bitmlx-quantized-4bitmlxllama-cpp
Symptom
Four records in data/benchmarks/ cite format_id '2bit' or 'mlx-quantized-4bit', producing 4 dangling reference warnings under tools/validate.py.
Root cause
data/quantization/ stores concrete quantization schemes with explicit numerical scales and packaging formats (e.g. bitsandbytes-nf4, gptq-4bit, gguf-q4-k-m). The tokens '2bit' and 'mlx-quantized-4bit' are format family names or bit-width descriptors rather than concrete serialization schemes. In MLX, 4-bit quantization can use affine (group size 64 default), mxfp4, or nvfp4 modes. In llama.cpp, '2bit' in this benchmark represents Q1_0 (a 2-bit trellis format) rather than a unified format record.
Workaround
Retain '2bit' and 'mlx-quantized-4bit' as budgeted dangling format references rather than synthesizing dummy format records or forcing inaccurate mappings.
notes
CITING RECORDS: llamacpp-radeon-r9700-gfx1201-q1-0-decode-tok-s (cites 2bit), local-m4-max-70b-q4-decode (cites mlx-quantized-4bit), local-m5-max-70b-q4-decode (cites mlx-quantized-4bit), and local-m5-ultra-70b-q4-decode (cites mlx-quantized-4bit). WHAT MUST NOT BE DONE: Never stub a format that is really a family name or bit-width descriptor. Creating fake format records for '2bit' or 'mlx-quantized-4bit' creates vague definitions and pollutes quantization metadata with underspecified schemes. Never repoint them to unrelated formats (such as gguf-iq1-s or ternary for 2bit, or bitsandbytes-nf4 for mlx). WHAT A FUTURE WAVE WOULD NEED: Concrete format records for the actual implementations measured: for 2bit in llama.cpp, a dedicated quantization record for llama.cpp Q1_0; for mlx-quantized-4bit, specific MLX format records identifying the exact quantization mode (e.g. mlx-affine-4bit with group size 64) and verifying the exact MLX invocation parameters used in each benchmark before updating the format_id references.

Notes

CITING RECORDS: llamacpp-radeon-r9700-gfx1201-q1-0-decode-tok-s (cites 2bit), local-m4-max-70b-q4-decode (cites mlx-quantized-4bit), local-m5-max-70b-q4-decode (cites mlx-quantized-4bit), and local-m5-ultra-70b-q4-decode (cites mlx-quantized-4bit). WHAT MUST NOT BE DONE: Never stub a format that is really a family name or bit-width descriptor. Creating fake format records for '2bit' or 'mlx-quantized-4bit' creates vague definitions and pollutes quantization metadata with underspecified schemes. Never repoint them to unrelated formats (such as gguf-iq1-s or ternary for 2bit, or bitsandbytes-nf4 for mlx). WHAT A FUTURE WAVE WOULD NEED: Concrete format records for the actual implementations measured: for 2bit in llama.cpp, a dedicated quantization record for llama.cpp Q1_0; for mlx-quantized-4bit, specific MLX format records identifying the exact quantization mode (e.g. mlx-affine-4bit with group size 64) and verifying the exact MLX invocation parameters used in each benchmark before updating the format_id references.

Cross-references

This record references no other records — it is a leaf. It is cited by 0.