Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

Leech-lattice quantization gives good quality at 2 bits/weight, but the codebooks have more than 10^14 points - far too many for a lookup table, so earlier kernels read 4.8 bits/weight from memory to store 2 bits of code.

A NEW codebook on the same Leech lattice, designed for lookup-table decoding: a 24-weight block still takes 48 bits, most of which index a 64-state TRELLIS of the Golay code plus one shared 16 KiB table. The kernel decodes a block with SIX TABLE LOADS AND TWO SMALL LOOKUPS inside the matvec and reads 2.148 bits per weight. For full models, retrain one scale per matrix row, store the matrices that lose the most as 4-bit integers, and pay for them with 4-bit embedding tables.

Paper's own claim (abstract, ALL numbers on ONE NVIDIA L40S): Qwen3-4B/8B/14B whole-model figures of 2.73 / 2.70 / 2.73 bits per parameter; MMLU (full test set) 63.37 / 69.58 / 75.66, i.e. 4.76 / 4.21 / 2.46 points BELOW 4-bit AWQ which runs at 5.3-6.0 bits/parameter; generate 113.8 / 95.0 / 57.2 tok/s in the authors' own engine; on GSM8K through the served kernel they lose 9.63 / 4.62 / 3.26 points to FP16. COMPRESSION RATIO: ~2.7 bits/parameter whole-model (~5.9x vs FP16). QUALITY COST: 2.46-4.76 MMLU points below AWQ, and up to 9.63 GSM8K points below FP16 at 4B - this is an honest, unusually complete quality statement.

arXiv ID 2609.35465 verified via /abs/ (title matches exactly). Venue: Semantic Scholar was rate-limiting and returned no record; the arXiv comment field is empty - arxiv-preprint, year = v1 year 2026, confidence lowered. The paper states the experiments were PREREGISTERED and that every table/figure number comes from one NVIDIA L40S, which is unusually good practice for a quantization claim and is why the numbers above are quoted verbatim. The tok/s figures are from the authors' own engine and are NOT comparable to vLLM or llama.cpp. arXiv ID and title verified by fetching https://arxiv.org/abs/2609.35465 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2609.35465) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.