dropbox/hqq GitHub README

The group size and backend matrix: recommended start is nbits=4, group_size=64, axis=1. nbits supports 8/4/3/2/1. axis=0 generally gives better quality but is NOT supported by the optimized inference runtimes (ATEN backend is axis=0 only). Optimized backends: gemlite (nbits 4/2/1, fp16, axis=1), torchao tiny_gemm int4 (nbits 4, bf16, axis=1), PYTORCH_COMPILE for fullgraph torch.compile. vLLM on-the-fly HQQ quantizers incl. mxfp8_dynamic, mxfp4_dynamic, nvfp4_dynamic. Quote: ~158 tok/s for Llama3-8B 4-bit on an RTX 4090. | SOURCE AUDIT 2026-10-03: the 'published' field held a verbatim copy of this notes paragraph instead of a date (field corruption across the qlab2-* batch); published set to null because a GitHub README, doc page or source header has no reliable publication date of its own. Content of the document itself was re-verified live 2026-10-03 and the notes are correct.