BitNet: Scaling 1-bit Transformers for Large Language Models

1-bit weights for LLMs had not been shown to scale with a normal power law or to be stable in training.

BitLinear: a drop-in replacement for nn.Linear in which the weight matrix is constrained to binary (-1/+1) by quantization-aware training and the activations are quantized to 8 bits; subtraction of the weight mean restores the activation distribution, and the projection's output is re-scaled after summation. Trained from scratch - this is a MODEL change, not a compression of a trained model, which is the crucial distinction.

Paper's own claim (abstract): 'competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines', and exhibits a scaling law akin to full-precision Transformers. NO compression ratio paired with a quality delta is given - BitNet is not a post-hoc compressor, so there is no baseline model being degraded, and the speedup_reported field stays qualitative rather than inventing a throughput number.

arXiv ID 2310.11453 verified via /abs/. Venue: Semantic Scholar returns arXiv.org only and the arXiv comment field is empty, so arxiv-preprint with year = v1 year 2023. This is the b1.0 BitNet, distinct from the b1.58 ternary paper (BitNet-b158). arXiv ID and title verified by fetching https://arxiv.org/abs/2310.11453 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2310.11453) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.