inference/research
verifiedmedium · 0.75updated 2026-10-03

FLUTE (lookup-table engine for LUT-quantized LLMs)

non-uniform 3-bit weight-only (also applied as an extension of NormalFloat quantization); activations fp16

Fields

id
quantization/flute
Scheme
flute
Bits
non-uniform 3-bit weight-only (also applied as an extension of NormalFloat quantization); activations fp16
Weight group size
128
Activation scheme
None - weight-only. The contribution is the KERNEL, not the format: offline restructuring of the quantized weight matrix to minimize unpacking bit-manipulation, plus vectorized and duplicated lookup tables to escape shared-memory bandwidth limits.
Kernels
FLUTE (Flexible Lookup Table Engine), Microsoft. Fuses dequantization into the matmul. Claimed 2-4x faster than existing GEMM kernels at batch < 32 with quantization group size 128, described as typical for LLM inference.
Quality delta
No direct quality number in the abstract. As an application of the engine the paper extends NormalFloat quantization and quantizes LLaMA3 'obtaining competitive quantization performance against strong baselines while obtaining an end-to-end throughput increase of 1.5 to 2x'. Quality is described only comparatively; none recorded.
notes
weight_group_size is 128 because the paper states it explicitly as the typical LLM inference configuration at which FLUTE's speedup is quoted - this is one of the few records in the repo where the group size is sourced rather than guessed. FLUTE is the right record for the argument that 3-bit weight-only is a KERNEL problem as much as a format problem: non-evenly-divisible bit widths like 3 bits make packing and unpacking dominate, and the fix is offline restructuring rather than a better grid. Enum mismatch: 'hqq' is nearest (non-uniform codebook quantization); the FLUTE engine is orthogonal to both NF4 and GPTQ and composes with either. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'hqq' (a proxy recorded while the enum lacked a legal value) and is now 'flute'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

weight_group_size is 128 because the paper states it explicitly as the typical LLM inference configuration at which FLUTE's speedup is quoted - this is one of the few records in the repo where the group size is sourced rather than guessed. FLUTE is the right record for the argument that 3-bit weight-only is a KERNEL problem as much as a format problem: non-evenly-divisible bit widths like 3 bits make packing and unpacking dominate, and the fix is offline restructuring rather than a better grid. Enum mismatch: 'hqq' is nearest (non-uniform codebook quantization); the FLUTE engine is orthogonal to both NF4 and GPTQ and composes with either. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'hqq' (a proxy recorded while the enum lacked a legal value) and is now 'flute'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: WEIGHT-ONLY, SO STRUCTURALLY ABSENT. The bits field keeps activations in FP16, so execution is dequantize-to-FP16 then an FP16 tensor-core GEMM; there is no INT4-x-INT4 (or sub-8-bit x sub-8-bit) silicon instruction for it to target. This is a statement about the arithmetic, not an absence of investigation. Consequence for a reader: nothing further to verify - no silicon path is possible for this format as defined. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)