Quantization formats (109)
Formats, bit widths and their hardware paths. Note native_support versus emulated_support — a format that only runs by dequantising to fp16 is not natively supported, whatever the vendor table implies.
GGUF Q3_K_M (3-bit K-quant, medium)
3.9960 measured avg (llama.cpp quantize README, Llama-3.1-8B). The pure block_q3_K is 3.4375 bpw (source comment); the measured 3.9960 is Q3_K_M, a MIXTURE of q3_K and wider types.Llama-3.1-8B-Instruct, one checkpoint, official llama.cpp toolchain, vs its own F16 baseline (quant-gguf-kquant-study): WikiText-2 ppl 7.96 vs 7.32 (+0.64); GSM8K 75.03 vs 77.63 (-2.60); HellaSwag 73.16 vs 72.51; IFEval 73.41 vs 78.93; MMLU…quantization/gguf-q3-k-m
contestedmedium · 0.82
GGUF Q4_K_M (4-bit K-quant, medium)
4.8944 measured avg (llama.cpp quantize README, Llama-3.1-8B). The pure block_q4_K is 4.5 bpw (source comment); 4.8944 is the Q4_K_M mixture.Llama-3.1-8B-Instruct (quant-gguf-kquant-study): WikiText-2 ppl 7.56 vs 7.32 F16 (+0.24, the smallest perplexity cost of any 4-bit option measured); GSM8K 69.41 vs 77.63; IFEval 72.35 vs 78.93; unweighted mean 69.15 vs 69.47 (-0.46% AvgLoss…quantization/gguf-q4-k-m
contestedhigh · 0.85
GGUF Q4_K_S (4-bit K-quant, small)
4.6672 measured avg (llama.cpp quantize README, Llama-3.1-8B); pure block_q4_K is 4.5 bpwLlama-3.1-8B-Instruct (quant-gguf-kquant-study): WikiText-2 ppl 7.62 vs 7.32 F16 (+0.30); GSM8K 70.83 vs 77.63; HellaSwag 77.33 vs 72.51; IFEval 72.79 vs 78.93; MMLU 80.26 vs 63.50; TruthfulQA 53.40 vs 54.79; unweighted mean 69.17 vs 69.47 …quantization/gguf-q4-k-s
contestedmedium · 0.80
GGUF Q5_K_M (5-bit K-quant, medium)
5.7036 measured avg (llama.cpp quantize README, Llama-3.1-8B); pure block_q5_K is 5.5 bpwLlama-3.1-8B-Instruct (quant-gguf-kquant-study): WikiText-2 ppl 7.40 vs 7.32 F16 (+0.08); GSM8K 64.35 vs 77.63 (-13.28, the study's clearest multi-step-reasoning regression among the 5-bit tier); IFEval 72.33 vs 78.93; unweighted mean 69.36…quantization/gguf-q5-k-m
contestedmedium · 0.82
GGUF Q6_K (6-bit K-quant)
6.5633 measured avg (llama.cpp quantize README, Llama-3.1-8B); block_q6_K is 6.5625 bpw (source comment) - Q6_K is essentially a pure typeLlama-3.1-8B-Instruct (quant-gguf-kquant-study): WikiText-2 ppl 7.35 vs 7.32 F16 (+0.03, within noise); unweighted mean 69.23 vs 69.47; size 6283 MiB (-58.98%). GSM8K reported 58.98 vs 77.63 for F16, which is a 18-point drop inconsistent wi…quantization/gguf-q6-k
contestedhigh · 0.85
QuaRot (rotation-based outlier removal, W4A4KV4)
w4a4kv4 (weights, activations AND KV cache all at 4 bits); the paper also reports lossless 6-bit and 8-bit round-to-nearest results with no calibration dataCONTESTED between two primary artifacts of the same paper. The arXiv v2 abstract states the 4-bit LLaMa2-70B model has 'losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance'; the repository README states…quantization/quarot
contestedmedium · 0.75
BitNet b1.58 (ternary LLM)
1.58 avgFrom the paper, as a training result rather than a post-training delta: BitNet b1.58 'matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task perf…quantization/bitnet-b158
draftlow · 0.60
EXL2 (GPTQ with mixed per-tensor bitrate)
2-8 adjustable (target average bits-per-weight, e.g. 2.5 / 3.0 / 4.0 / 5.0 bpw builds)quantization/exl2
draftlow · 0.60
FP8 KV cache quantization (E4M3 keys and values)
8-bit KV cache (E4M3 format for both keys and values)No controlled benchmark with named accuracy delta found in the sources accessed. The vLLM quantization docs list 'Quantized KV Cache' as a supported feature but do not publish accuracy figures. The SGLang docs mention FP8 KV cache support b…quantization/qhw-fp8-kv-cache
draftlow · 0.50
INT8 KV cache quantization (8-bit keys and values)
8-bit KV cache (INT8 format for both keys and values)quantization/qhw-int8-kv-cache
draftlow · 0.50
MXFP4 MoE quantization: per-expert scales and block scaling for mixture-of-experts
w4a4 (MXFP4 weights and activations for MoE layers)quantization/qhw-moe-mxfp4
draftlow · 0.50
MXFP6 (OCP microscaling FP6, E3M2/E2M3 + E8M0)
6-bit elements in 32-value blocks with a shared power-of-two scale; used as one channel of a mixed-precision MX pipeline, typically alongside MXFP4 and MXFP8None recorded. The source paper states 'competitive or superior performance across diverse downstream tasks, including zero-shot and few-shot learning, language modeling, code generation, and mathematical reasoning' without a named numeric …quantization/mxfp6
draftlow · 0.50
QuARX (GPU-efficient 4-bit-weight / 8-bit-activation quantization) - UNVERIFIED PLACEHOLDER
w4a8 (believed; not confirmed against a primary source)quantization/quarx
draftlow · 0.15
SmoothQuant with FP8 activations: does outlier migration still help?
w8a8 (8-bit weights, 8-bit activations)quantization/qhw-smoothquant-fp8-interaction
draftlow · 0.40
Accuracy cost of quantization: measured deltas on named benchmarks
varies by methodSUMMARY OF MEASURED ACCURACY DELTAS: (1) Marlin INT4 (Llama2-7B, WikiText-2): 5.12 (FP16) vs 5.27 (INT4) = +0.15 ppl. MMLU: 41.80 vs 40.07 = -1.73. (2) GPTQ 3-bit (Llama-7B, WikiText-2): 5.68 (FP16) vs 8.07 (3-bit) = +2.39 ppl. With group-1…quantization/qhw-accuracy-cost-summary
verifiedhigh · 0.85
AQLM (additive / multi-codebook quantization)
2, 3 and 4-bit weights (targeting sub-4-bit); activations remain fp16, so this is w2a16 / w3a16From the paper's abstract: AQLM is 'the first scheme that is Pareto optimal in terms of accuracy-vs-model-size when compressing to less than 3 bits per parameter' and 'significantly improves upon all known schemes in the extreme compression…quantization/aqlm
verifiedmedium · 0.80
Atom (mixed-precision block-outlier low-bit quantization for serving)
w4a4 (4-bit weights AND 4-bit activations) with mixed-precision outlier channels retained at higher precisionFrom the abstract: end-to-end throughput up to 7.73x versus FP16 and 2.53x versus INT8 quantization at the same latency target, with negligible accuracy loss, in the serving context. These are throughput figures from the authors' own system…quantization/atom
verifiedmedium · 0.80
AutoRound (Intel) - SignRound weight-rounding PTQ toolkit, W4A16 and neighbours
W2A16 / W3A16 / W4A16 / W5A16 / W6A16 / W7A16 / W8A16, all defined as bits=N, group_size=128, sym=True, act_bits=16. Plus NVFP4 (group_size 16), MXFP4/MXINT4/MXFP4_RCEIL/MXFP8 (group_size 32, flagged 'research feature, no real kernel'), FP8_STATIC, FP8_BLOCK, FPW8A16 (group_size 0 = per-tensor)Vendor claim, one controlled number with a named model: the INT2-mixed DeepSeek-R1 (~200GB) retains 97.9% accuracy (OPEA/DeepSeek-R1-int2-mixed-sym-inc, qlab2-autoround-step-by-step). Methodology (benchmark, eval set, baseline) is not state…quantization/autoround-w4a16
verifiedhigh · 0.85
AWQ (activation-aware weight quantization)
w4a16From the AWQ paper itself, OPT-6.7B under INT3-g128: WikiText perplexity 43.2 for plain RTN vs 13.0 when the salient 1% of weights are identified by activation distribution and kept in FP16; AWQ's per-channel scaling is the hardware-efficie…quantization/awq
verifiedhigh · 0.85
bitsandbytes NF4 (4-bit NormalFloat)
w4a16quantization/bitsandbytes-nf4
verifiedmedium · 0.70
CacheGen (KV cache tensor encoding + bandwidth-adaptive streaming)
adaptive per-part compression level; the paper reports a 3.5-4.3x reduction in KV cache SIZE with negligible response-quality impactFrom the abstract: reduces KV cache size by 3.5-4.3x and total delay in fetching and processing contexts by 3.2-3.7x, with negligible impact on LLM response quality, versus systems that reuse the KV cache.quantization/cachegen
verifiedmedium · 0.78
DuoAttention (head-wise retrieval/streaming KV sparsity) - NOT KV quantization
none by default - retrieval heads keep a full KV cache, streaming heads use a lightweight CONSTANT-LENGTH KV cache. The paper's headline long-context result is achieved combined WITH quantization.From the abstract: reduces long-context inference memory by up to 2.55x for MHA and 1.67x for GQA, speeds decoding by up to 2.18x (MHA) and 1.50x (GQA), and accelerates pre-filling by up to 1.73x (MHA) and 1.63x (GQA), with minimal accuracy…quantization/duoattention-kv-sparsity
verifiedmedium · 0.80
EXL3 (ExLlamaV3 format, QTIP-based trellis quantization) - the successor to the archived EXL2
target average bpw, fractional (e.g. 2.0-8.0; benchmark chart spans 1.75-8.0)quantization/exl3
verifiedmedium · 0.80
ExLlamaV2 EXL2 kernel throughput benchmarks (3090 Ti / 4090)
2-8 bpw (adjustable average bits-per-weight)quantization/qhw-exllama-throughput
verifiedhigh · 0.85
FLUTE (lookup-table engine for LUT-quantized LLMs)
non-uniform 3-bit weight-only (also applied as an extension of NormalFloat quantization); activations fp16No direct quality number in the abstract. As an application of the engine the paper extends NormalFloat quantization and quantizes LLaMA3 'obtaining competitive quantization performance against strong baselines while obtaining an end-to-end…quantization/flute
verifiedmedium · 0.75
FP8 block scaling (blockwise, E4M3 with per-block scale)
w8a8, block-scaledquantization/fp8-blockwise
verifiedmedium · 0.70
FP8 E4M3 (per-tensor / per-block scaled)
w8a8 (fp8 weights, fp8 activations, fp32 accumulate)No controlled fp16 delta was located in the primary sources read for this record. Vendor checkpoint claims (Llama-3.1-8B-Instruct-FP8 et al. shipped by NVIDIA) exist but are not a perplexity comparison, so none recorded.quantization/fp8-e4m3
verifiedhigh · 0.85
FP8 E5M2
w8a8 (or, in practice, KV-cache format)quantization/fp8-e5m2
verifiedmedium · 0.75
GGUF imatrix (importance matrix) - the calibration file that determines WHICH weights get more bits
not a format - a per-tensor activation-energy file. It changes the effective bits of an existing quant choice, not the nominal bits of a typen/a - the imatrix is a calibration artifact, not a quantization format, so it has no quality_delta of its own. Its effect is to REMOVE part of the quantization delta of the format it is applied to. IMPORTANT: the official per-type ppl strin…quantization/gguf-imatrix
verifiedhigh · 0.85
GGUF IQ1_M (1.75 bpw trellis-flavoured 1-bit with 3-bit sub-block scales and a grid-shift bit)
2.1460 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '1.75 bpw quantization'. block_iq1_m = { uint8_t qs[QK_K/8]; uint8_t qh[QK_K/16]; uint8_t scales[QK_K/32]; } = 32 + 16 + 8 = 56 bytes / 256 = 1.75 bpwOfficial tool string '1.75 bpw quantization' (qlab2-quantize-options-table); no controlled fp16 delta published. Null + note.quantization/gguf-iq1-m
verifiedmedium · 0.80
GGUF IQ1_S (1.5625 bpw, non-linear 1-bit-ish with a 16-bit grid high plane)
2.0042 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '1.56 bpw quantization'. block_iq1_s = { ggml_half d; uint8_t qs[QK_K/8]; uint16_t qh[QK_K/32]; } = 2 + 32 + 16 = 50 bytes / 256 = 1.5625 bpwOfficial tool string '1.56 bpw quantization' (qlab2-quantize-options-table); no controlled fp16 delta published. Null + note.quantization/gguf-iq1-s
verifiedmedium · 0.80
GGUF IQ2_M (mixture built on IQ2_S with Q4_K/Q5_K promoted tensors)
2.9294 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '2.7 bpw quantization'. NOT a distinct block type - llama_tensor_get_quant_type maps LLAMA_FTYPE_MOSTLY_IQ2_M to GGML_TYPE_IQ2_SOfficial tool string '2.7 bpw quantization' (qlab2-quantize-options-table); no controlled fp16 delta published. Null + note.quantization/gguf-iq2-m
verifiedmedium · 0.80
GGUF IQ2_S (2.5625 bpw, non-linear 2-bit with a 4-bit scale per 64 and sign/high-bit planes)
2.7403 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '2.5 bpw quantization'. block_iq2_s = { ggml_half d; uint8_t qs[QK_K/4]; uint8_t qh[QK_K/32]; uint8_t scales[QK_K/32]; } = 2 + 64 + 8 + 8 = 82 bytes / 256 = 2.5625 bpwOfficial tool string '2.5 bpw quantization' for IQ2_S and '2.7 bpw quantization' for IQ2_M (qlab2-quantize-options-table); no controlled fp16 delta published. Null + note.quantization/gguf-iq2-s
verifiedmedium · 0.80
GGUF IQ2_XS (2-bit non-linear, 256 super-block with a 32-element sub-block scale plane)
2.3125 bpw (ggml header: '2.3125 bpw quants')quantization/gguf-iq2-xs
verifiedhigh · 0.85
GGUF IQ2_XXS (2.0625 bpw, almost-true 2-bit with a 16-bit scale per 256)
2.3824 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '2.06 bpw quantization'. block_iq2_xxs = { ggml_half d; uint16_t qs[QK_K/8]; } = 2 + 64 = 66 bytes / 256 = 2.0625 bpwOfficial tool string '2.06 bpw quantization' (qlab2-quantize-options-table); no controlled fp16 delta published. Null + note.quantization/gguf-iq2-xxs
verifiedmedium · 0.80
GGUF IQ3_S (3.4375 bpw, non-linear 3-bit with separate sign plane)
3.6606 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '3.44 bpw non-linear quantization'. block_iq3_s = { ggml_half d; uint8_t qs[QK_K/4]; uint8_t qh[QK_K/32]; uint8_t signs[QK_K/8]; uint8_t scales[IQ3S_N_SCALE]; } with IQ3S_N_SCALE = QK_K/64 = 4Official tool string '3.44 bpw non-linear quantization' for IQ3_S and '3.66 bpw quantization mix' for IQ3_M (qlab2-quantize-options-table). No controlled fp16 delta published. Null + note.quantization/gguf-iq3-s
verifiedmedium · 0.80
GGUF IQ3_XXS (3.0625 bpw, almost-true 3-bit with a 16-bit scale per 256)
3.2548 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '3.06 bpw quantization'. block_iq3_xxs = { ggml_half d; uint8_t qs[3*QK_K/8]; } = 2 + 96 = 98 bytes / 256 = 3.0625 bpwOfficial tool string '3.06 bpw quantization' (qlab2-quantize-options-table); no controlled fp16 delta published. Null + note.quantization/gguf-iq3-xxs
verifiedmedium · 0.80
GGUF IQ4_NL (non-linear 4-bit, 32-value blocks, QK4_NL)
4.6818 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '4.50 bpw non-linear quantization'. Block layout: QK4_NL = 32, block_iq4_nl = { ggml_half d; uint8_t qs[QK4_NL/2]; } = 18 bytes / 32 = 4.5 bpwOfficial tool string only: IQ4_XS '4.25 bpw non-linear quantization' and IQ4_NL '4.50 bpw non-linear quantization' (qlab2-quantize-options-table). No controlled fp16 delta is published for IQ4_NL in any primary source found; the llama.cpp q…quantization/gguf-iq4-nl
verifiedmedium · 0.80
GGUF IQ4_XS (non-linear 4-bit, 256-value super-block with 6-bit sub-block scales)
4.4597 measured avg (llama.cpp quantize README, Llama-3.1-8B); official tool string '4.25 bpw non-linear quantization'. block_iq4_xs = { ggml_half d; uint16_t scales_h; uint8_t scales_l[QK_K/64]; uint8_t qs[QK_K/2]; } = 2 + 2 + 4 + 128 = 136 bytes / 256 = 4.25 bpwOfficial tool string '4.25 bpw non-linear quantization' (qlab2-quantize-options-table); no controlled fp16 perplexity delta published in any primary source found. Null + note.quantization/gguf-iq4-xs
verifiedmedium · 0.80
GGUF MXFP4_MOE (llama.cpp MoE-specific ftype: MXFP4 for experts, Q8_0 for everything else)
MXFP4 (4-bit E2M1 + shared E8M0 scale) on expert tensors, Q8_0 on non-expert tensorsquantization/gguf-mxfp4-moe
verifiedhigh · 0.85
GGUF Q2_K / Q2_K_S (2-bit K-quant)
2.625 effective bpw for block_q2_K (source comment); 2.9697 measured avg for Q2_K_S and 3.1593 for plain Q2_K on Llama-3.1-8B (llama.cpp quantize README)Only the unified Llama-3.1-8B-Instruct study was found for the 2-bit K-quant tier. The closest measured point is Q3_K_S (3.9960 bpw, same family): WikiText-2 ppl 8.96 vs 7.32 FP16, GSM8K 68.31 vs 77.63, IFEval 71.87 vs 78.93, unweighted 5-b…quantization/gguf-q2-k
verifiedhigh · 0.85
GGUF Q3_K_L (3-bit K-quant, large)
4.1200 measured avg (llama.cpp quantize README, Llama-3.1-8B). Pure block_q3_K is 3.4375 bpw; Q3_K_L is a mixture promoting attention and feed-forward tensors to Q5_K.Llama-3.1-8B-Instruct vs FP16 baseline from Table 2 of Kurt (2026, https://arxiv.org/abs/2601.14277, quant-gguf-kquant-study): WikiText-2 perplexity 7.81 vs 7.32 (+0.49 ppl delta); downstream 5-benchmark average 68.78 vs 69.47 (-0.69 delta)…quantization/gguf-q3-k-l
verifiedhigh · 0.88
GGUF Q3_K_S (3-bit K-quant, small)
3.4375 bpw base (block_q3_K = 110 bytes / 256 values); 3.9960 measured avg on Llama-3.1-8B (llama.cpp quantize README). Small variant emphasizing compression.Llama-3.1-8B-Instruct vs FP16 baseline from Table 2 of Kurt (2026, https://arxiv.org/abs/2601.14277, quant-gguf-kquant-study): WikiText-2 perplexity 8.96 vs 7.32 (+1.64 ppl delta); downstream 5-benchmark average 65.49 vs 69.47 (-3.98 delta)…quantization/gguf-q3-k-s
verifiedhigh · 0.88
GGUF Q4_0 (legacy block-4 symmetric)
4.5 (classic block format; Q4_0 measured 4.4597 bpw in the IQ sweep, Q4_0 as a legacy 4-bit type is quoted as ~4)Llama-3.1-8B-Instruct (quant-gguf-kquant-study): WikiText-2 ppl 7.74 vs 7.32 F16; GSM8K 71.03 vs 77.63 (-6.60); IFEval 71.88 vs 78.93; mean 67.98 vs 69.47; size 4438 MiB (-71.03%).quantization/gguf-q4-0
verifiedmedium · 0.78
GGUF Q4_1 / Q5_1 (legacy asymmetric 32-value blocks with per-block scale AND min)
4.5 bpw for q4_1 (QK4_1 = 32: 2 x fp16 + 16 = 36 bytes / 32 values), 5.5625 bpw for q5_1 (2 x fp16 + 4 + 16 = 40 bytes / 32)Official quantizer strings: Q4_0 '4.34G, +0.4685 ppl @ Llama-3-8B', Q4_1 '4.78G, +0.4511 ppl', Q5_0 '5.21G, +0.1316 ppl', Q5_1 '5.65G, +0.1062 ppl', Q8_0 '7.96G, +0.0026 ppl' (qlab2-quantize-options-table). All are the tool's own Llama-3-8B…quantization/gguf-legacy-q4-1
verifiedmedium · 0.80
GGUF Q8_0 (legacy block-8 symmetric)
8.5008 avg (llama.cpp measured, Llama-3.1-8B)Llama-3.1-8B-Instruct (quant-gguf-kquant-study): WikiText-2 ppl 7.33 vs 7.32 F16 (+0.01); unweighted mean 69.41 vs 69.47 (-0.09% AvgLoss); size 8138 MiB (-46.87%). This is the reference-ish GGUF baseline: 8.5008 bpw includes block scales, s…quantization/gguf-q8-0
verifiedmedium · 0.80
GGUF TQ1_0 / TQ2_0 (ternarized quants, QK_K super-block, trit-packed)
TQ1_0 1.6875 bpw, TQ2_0 2.0625 bpw (both from the struct comments in ggml-common.h). Quantize tool strings: TQ1_0 '1.69 bpw ternarization', TQ2_0 '2.06 bpw ternarization'Official quantizer strings only (qlab2-quantize-options-table): TQ1_0 '1.69 bpw ternarization', TQ2_0 '2.06 bpw ternarization'. No controlled fp16 perplexity delta published in any primary source found. Null + note.quantization/gguf-tq1-0
verifiedmedium · 0.80
GPTQ (one-shot approximate second-order PTQ)
w4a16 headline; also reported at w3a16 (and 2-bit / ternary) in the paper's extreme regime. Weight-only - activations are NOT quantized.From the GPTQ paper: OPT-175B and BLOOM-176B quantized to 3-4 bits in ~4 GPU hours with 'minimal increase in perplexity'; the bespoke execution kernels give ~3.25x on A100 and 4.5x on A6000 (quant-gptq-paper). The paper's Figure 1 is a ppl-…quantization/gptq
verifiedhigh · 0.85
H2O (heavy-hitter KV eviction) - NOT KV quantization
none - no bits are reduced. Evicts tokens instead. Retains a configurable budget of heavy-hitter plus recent tokens (20% heavy hitters in the paper's throughput results).From the abstract: with 20% heavy hitters, H2O improves throughput over DeepSpeed Zero-Inference by up to 29x, HuggingFace Accelerate by up to 29x, and FlexGen by up to 3x on OPT-6.7B and OPT-30B; at the same batch size it reduces latency b…quantization/h2o-kv-eviction
verifiedmedium · 0.80
HQQ (half-quadratic quantization)
1-8: nbits supports 8, 4, 3, 2 and 1 (qlab2-hqq-readme). Commonly 2/3/4.Only a cross-model claim, not a controlled fp16 delta: HQQ-quantized Llama-2-70B at 2-bit is reported as outperforming full-precision Llama-2-13B at comparable memory (quant-hqq-blog). The comparable controlled claim is speed: HQQ quantizes…quantization/hqq
verifiedmedium · 0.70
InfiniGen (speculative KV prefetch for offloaded serving) - NOT KV quantization
none - no bits reduced. Reduces how much of the KV cache must be FETCHED from host memory.From the abstract: improves overall performance of a modern offloading-based system by up to 3.00x compared to prior KV cache management methods, while offering 'substantially better model accuracy'. No accuracy number given; none recorded.quantization/infinigen-kv
verifiedmedium · 0.78
INT8 W8A8 (weight-only scheme, per-tensor / per-channel)
w8a8quantization/int8-w8a8
verifiedmedium · 0.70
INT8 W8A8 rowwise / per-channel
w8a8quantization/int8-channelwise
verifiedmedium · 0.65
Intel Neural Compressor (multi-technique PTQ/QAT toolkit; low-bit LLM path is AutoRound)
the toolkit spans INT8 / FP8 / MXFP8 / INT4 / MXFP4 / NVFP4 plus sparsity; its own advanced low-bit LLM/VLM quantization is DELEGATED to AutoRoundn/a - the README cites 'Faster Inference of LLMs using FP8 on the Intel Gaudi' (arXiv:2503.09975) but does not restate a delta inline. Null + note rather than paraphrasing a paper I did not read for this record.quantization/neural-compressor-stack
verifiedhigh · 0.85
KIVI (2-bit key / 4-bit value KV cache, per-channel keys, per-token values)
2-bit keys / 4-bit values - ASYMMETRIC, and the asymmetry is the paper's central findingFrom the abstract: KIVI lets Llama, Falcon and Mistral maintain 'almost the same quality' while using 2.6x less peak memory (INCLUDING model weight), which enables up to 4x larger batch size and brings 2.35x-3.47x throughput on a real LLM i…quantization/kivi-kv2
verifiedhigh · 0.85
KVQuant (per-channel + pre-RoPE key quantization, non-uniform KV)
3-bit keys and values are the headline (<0.1 perplexity degradation); the method is a toolkit that also supports 2-bit and 4-bitFrom the abstract: '<0.1 perplexity degradation with 3-bit quantization on both Wikitext-2 and C4', outperforming existing approaches, across LLaMA, Llama-2, Llama-3 and Mistral. Enables serving LLaMA-7B at up to 1 million context on a sing…quantization/kvquant
verifiedhigh · 0.85
LLM-FP4 (4-bit floating-point W4A4 PTQ, pre-Blackwell)
w4a4 (both weights and activations quantized to 4-bit FLOATING POINT, post-training)From the abstract: quantizes both weights and activations of LLaMA-13B to 4-bit, achieving an average score of 63.1 on commonsense zero-shot reasoning tasks, 'only 5.8 lower than the full-precision model' and 'significantly outperforming th…quantization/llm-fp4
verifiedmedium · 0.80
LLM-QAT (data-free quantization-aware training, NVIDIA)
w4a4 to w8a8 plus 4-bit KV cache (per-tensor/per-channel groups configured per experiment)Paper states 'large improvements over training-free methods, especially in the low-bit settings' for LLaMA 7B/13B/30B down to 4-bit. The abstract gives no numeric perplexity delta and the experiments section was not read this session, so no…quantization/llm-qat
verifiedmedium · 0.80
LLM-QAT KV cache quantization (4-bit KV inside a QAT regime)
4-bit KV cache, alongside quantized weights and activationsquantization/llm-qat-kv-cache
verifiedmedium · 0.70
LLM.int8() / bitsandbytes int8 (vector-wise with 16-bit outliers)
w8a16 (mixed-precision decomposition: >99.9% of values multiplied in 8-bit, outlier dims in fp16)From the primary paper (qlab-llmint8-paper): a 175B-parameter 16/32-bit checkpoint can be loaded, converted to Int8 and used immediately with no performance degradation, because the mixed-precision decomposition keeps 'more than 99.9% of va…quantization/bitsandbytes-int8
verifiedmedium · 0.75
llmcompressor (Red Hat AI / vLLM project) + compressed-tensors checkpoint format
the scheme set is named, not numeric: FP8_BLOCK, FP8_DYNAMIC, INT8, W2A16..W8A16 (and G32/G64 group variants), MXFP4, MXFP8, NVFP4, plus arbitrary-bit and mixed-precision. The documented worked example is FP8 weights with RTN at block_size 128 and dynamic FP8 activationsn/a - the README ships no controlled fp16 delta. The documented example uses Qwen/Qwen3-30B-A3B and only says 'Confirm generations of the quantized model look sane'. Null + note.quantization/llmcompressor-quant
verifiedhigh · 0.85
LQER (low-rank quantization error reconstruction)
w4a8 (4-bit weights, 8-bit activations) - claimed near-losslessFrom the abstract: 'near-lossless W4A8 quantization on various LLMs and downstream tasks' without knowledge distillation, grid search, or gradient iterative optimization; near-lossless on six downstream tasks while using 1.36x fewer hardwar…quantization/lqer
verifiedmedium · 0.75
Machete Hopper-optimized mixed-precision GEMM kernel (CUTLASS-based)
w4a16 (4-bit weights, FP16 activations) - same precision as Marlin but optimized for Hopper SM90quantization/qhw-machete-kernel
verifiedmedium · 0.80
Marlin INT4 weight-only kernel (FP16xINT4 mixed-precision GEMM)
w4a16 (4-bit weights, FP16 activations)From the Marlin README benchmark table (Llama2 models, INT4 group=128): 7B WikiText-2 5.12 (FP16) vs 5.27 (INT4), RedPajama 6.14 vs 6.30, MMLU 41.80 vs 40.07. 13B: WikiText-2 4.57 vs 4.67, RedPajama 5.67 vs 5.79, MMLU 52.10 vs 51.13. 70B: W…quantization/qhw-marlin-kernel
verifiedhigh · 0.90
Microsoft Olive (Onnx LIVE) - hardware-targeted ONNX optimization / quantization composer
not a single format: Olive runs search over conversion + quantization passes and emits whichever ONNX quant configuration wins for the declared target. Quantization algorithms were expanded in Olive 0.8.0 (2025-03) and again in 0.9.2 (2025-08)n/a - no quality delta published. Null + note.quantization/olive-quant-stack
verifiedmedium · 0.70
MLC LLM compiler quantization (q4f16 group-quant, plus q3f16, AWQ, FP8 per-tensor and FP8 block-scale)
q4f16_* = int4 weights / fp16 activations with group_size 32; q3f16_* = int3 with group_size 40; q4f16_autoawq = AWQ with group_size 128; q4f16_ft = faster-transformer int4-in-int8; e4m3_e5m2_f16 = FP8 W8A8 per-tensor; fp8_e4m3fn_bf16_block_scale = FP8 blockwisen/a - MLC publishes no controlled fp16 quality delta in the registry or the intro docs. Null + note.quantization/mlc-quant-q4f16
verifiedmedium · 0.80
MLX affine quantized linear (mlx-lm's default 4-bit weight format, group_size 64)
w4a16 default (mlx affine mode also supports 2, 3, 4, 5, 6, 8 bits and per-group sizes; activations stay fp16/bf16)quantization/mlx-affine-quantized
verifiedhigh · 0.90
MoE mixed-precision expert bit allocation (expert-level bit-width, not one fixed precision)
per-expert or per-expert-group bit widths. The published endpoints span 2.25-bit (MxMoE), ~2-bit (MoQE, AlphaQ, EAQuant W2A4) and 4-bit W8A8-style expert weights; AutoScheme generates mixed-bit recipes in minutesThe only figures in this cluster with a named model, benchmark and delta are from MxMoE (qlab2-arxiv-mxmoe): 2.4 lower Wikitext-2 perplexity than GPTQ at 2.25-bit, up to 3.4x speedup over full precision, and up to 29.4% speedup over uniform…quantization/moe-mixed-precision
verifiedmedium · 0.75
MoE routing-instability under quantization (VSRAQ / EAQuant routing-consistency alignment)
not a distinct format - a PTQ OBJECTIVE that constrains which bit-allocations are acceptable, used on top of otherwise-standard schemes at W4A4 / W3A4 / W3A3 / W2A4EAQuant reports average accuracy improvements of 1.15-13.81% over existing methods across three MoE architectures at W4A4/W3A4/W3A3/W2A4, with the largest gains on reasoning tasks (qlab2-arxiv-eaquant). Architectures and benchmark names are…quantization/moe-routing-quant-interaction
verifiedmedium · 0.70
MXFP4 (OCP microscaling FP4, E8M0 scale)
w4a4No inference fp16 delta recorded from a controlled benchmark. The closest primary evidence is a PTQ study of SmoothQuant+AWQ+GPTQ applied to MX formats, which reports 4-bit weights with 8-bit activations in MXINT with 'negligible accuracy l…quantization/mxfp4
verifiedmedium · 0.75
MXFP4 vs NVFP4: block size, scale format, and hardware support comparison
w4a4 (both weights and activations quantized to 4-bit floating point)NVIDIA's own comparison (from the NVFP4 blog) rates MXFP4 as carrying 'risk of noticeable accuracy drop compared to FP8' where NVFP4 does not. The scale-encoding MSE is 0.72 for E8M0 (MXFP4) vs 0.08 for E4M3 (NVFP4). This is a vendor rankin…quantization/qhw-mxfp4-vs-nvfp4
verifiedmedium · 0.80
MXFP8 (OCP microscaling FP8, E4M3 + E8M0)
w8a8, block-scaledquantization/mxfp8
verifiedmedium · 0.70
NVFP4 (Blackwell block-scaled E2M1 + E4M3)
w4a4 (fp4 weights and fp4 activations with hardware-native block-scaled tensor-core path)NVIDIA, DeepSeek-R1-0528, FP8 vs NVFP4, seven benchmarks (vendor-reported): MMLU-Pro 85% vs 84%, GPQA Diamond 81% vs 80%, HLE 15% vs 14%, LiveCodeBench 77% vs 76%, SciCode 40% vs 40%, Math-500 98% vs 98%, AIME 2024 89% vs 91%. The blog stat…quantization/nvfp4
verifiedmedium · 0.80
NVIDIA Model Optimizer (ModelOpt) quantization recipes
w4a16 / w4a8 / w8a8 / w4a4-fp4 depending on the recipe (FP4, FP8 per-tensor, FP8 block scaling, FP8 rowwise, NVFP4, W4A16/W4A8 AWQ, W4A16/W4A8 GPTQ)quantization/modelopt
verifiedmedium · 0.75
OmniQuant (learnable weight clipping + equivalent transformation)
reported at w2a16, w3a16, w4a16, w4a4, w6a6 and w8a8 (paper enumerates these configurations)Paper states LLaMA-2 7B-70B is processed within 1-16 hours on a single A100-40G using 128 samples, and validates across w2a16/w3a16/w4a16/w4a4/w6a6 without naming a headline accuracy number in the abstract. No numeric quality delta recorded…quantization/omniquant
verifiedmedium · 0.80
ONNX Runtime GenAI model-builder quantization paths (int4, fp8, ModelOpt/compressed-tensors ingest)
int4 (MatMulNBits weight-only, block size selectable), fp16/bf16/fp32 I/O, plus direct ingest of an already-quantized AutoGPTQ/AutoAWQ checkpoint and of a ModelOpt or compressed-tensors NVFP4/FP8 checkpointn/a - the builder README documents no numeric quality delta. Null + note.quantization/ortgenai-int4-fp8
verifiedmedium · 0.80
ONNX Runtime INT4/UInt4 via MatMulNBits and GatherBlockQuantized (weight-only, blockwise)
4-bit weights (Int4 if is_symmetric=True, else UInt4). Block size must be a power of two >= 16; the documented example is block_size=128n/a - no numeric delta published by Microsoft. Null + note.quantization/ort-matmulnbits-int4
verifiedhigh · 0.90
ONNX Runtime quantization toolset (quantize_static / quantize_dynamic, QDQ vs QOperator)
8-bit weights and activations. Four sign combinations: U8U8, U8S8, S8U8, S8S8. reduce_range=True drops weights to 7 bits on non-VNNI AVX2/AVX512n/a - none published. The ORT docs deliberately describe the failure modes instead: 'it is not rare to get worse performance on old devices', and quantization is called 'not a loss-less transformation'. Null + this note.quantization/ort-quant-int8
verifiedhigh · 0.90
optimum-quanto (Hugging Face quantization library)
int2 / int4 / int8 / float8 weights (and optional int8/float8 activations)Qualitative only, from the project README: models compiled with int8/float8 weights and float8 activations are 'very close to the full-precision models', and inference latency with quantized-weights-only is comparable to full precision wher…quantization/optimum-quanto
verifiedmedium · 0.65
Pensieve (multi-tier conversation-state KV cache) - NOT KV quantization
none - no bits reduced. Retains and reuses conversation state across requests in a multi-tier cache.From the abstract: 1.14x-3.0x the throughput of vLLM and TensorRT-LLM with significantly reduced latency, in the multi-turn conversation setting.quantization/pensieve
verifiedmedium · 0.78
Per-engine per-format NATIVE-vs-EMULATED quantization KERNEL matrix (which kernel actually executes on which arch, and which fallback it reaches when none is native) — the layer underneath the engine support matrices
matrix record - spans NVFP4 W4A4/W4A16, MXFP4, MXFP6, MXFP8, FP8 per-tensor/block/rowwise, INT8 W8A8, W4A8-FP8, AWQ/GPTQ W4A16, and KV-cache dtypesquantization/w6q-kernel-matrix-native-vs-emulated
verifiedhigh · 0.90
PyramidKV (layer-dependent KV budget) - NOT KV quantization
none - allocates different KV cache SIZES per layer (more in lower layers, less in higher ones) rather than reducing bits.From the abstract: matches full-KV-cache performance on LongBench while retaining only 12% of the KV cache; at an extreme 0.7% retention it surpasses other KV compression techniques with up to 20.5 absolute accuracy improvement on TREC; ret…quantization/pyramidkv
verifiedmedium · 0.80
QServe / QoQ (W4A8KV4 quantization and system co-design)
w4a8kv4 (4-bit weights, 8-bit activations, 4-bit KV cache) - explicitly named QoQ, quattuor-octo-quattuor, i.e. 4-8-4Measured quality from Table 2 and Table 3 of the paper (arXiv:2405.04532, qlab-qserve-paper): WikiText-2 perplexity (seqlen 2048) on Llama-3-8B is 6.70 (W4A8KV4 g128) and 6.81 (W4A8KV4 per-channel) vs 6.14 FP16 baseline (+0.56 to +0.67 ppl …quantization/qserve-w4a8kv4
verifiedmedium · 0.80
QServe SmoothAttention (4-bit KV cache with an accuracy-preserving correction)
4-bit KV cache (keys and values), the 'KV4' of W4A8KV4Throughput figures only in the abstract (LLaMA-3-8B 1.2x on A100 / 1.4x on L40S vs TensorRT-LLM; Qwen1.5-72B 2.4x on A100 / 3.5x on L40S). No KV-specific quality number in the abstract.quantization/qserve-kv4
verifiedmedium · 0.75
Quantization KERNEL DISPATCH SHAPES across serving engines — five structurally different ways an engine picks the kernel that runs a format, which is why cross-engine format-support tables cannot be compared cell by cell
matrix record - spans GGUF k/i/MX quants, AWQ/GPTQ W4A16, FP8 block-scaled, MXFP4, MXFP6, INT8 and NVFP4quantization/w6q-engine-kernel-dispatch-shapes
verifiedhigh · 0.90
Quantization runtime-version gates per format per architecture (CUDA / FlashInfer / torch / TVM-build / optional-package), the axis no support matrix has a column for
matrix record - spans NVFP4, MXFP4, MXFP6, MXFP8, FP8 per-tensor, FP8 block-scaled, INT8, AWQ/GPTQ W4A16 and the KV-cache dtypesquantization/w6q-runtime-version-gates
verifiedhigh · 0.95
QuaRot KV cache 4-bit (rotated KV, no retained outlier channels)
4-bit KV cache (keys and values), obtained after Hadamard rotation; part of the w4a4kv4 configurationquantization/quarot-kv4
verifiedmedium · 0.70
Quest (query-aware KV page sparsity) - NOT KV quantization
none - no bits reduced. Only the top-K critical KV pages are LOADED for attention; the rest stay in memory but are not read.Measured quality from Tang et al. (ICML 2024, arXiv:2406.10774, qlab-quest-paper): On PG19 32k-context language modeling (LongChat-7B-v1.5-32k), Quest with token budget 4096 (1/8 cache) matches full-cache oracle perplexity within 0.01 ppl (…quantization/quest-kv-sparsity
verifiedmedium · 0.80
QuIP# (Hadamard incoherence processing + E8 lattice codebooks)
2, 3 and 4-bit weight-only (<=4 bits per weight); activations fp16The abstract claims 'state-of-the-art results in extreme compression regimes (<=4 bits per weight)', 'enables new behaviors in PTQ scaling', and 'supports fast inference', with no numeric perplexity in the abstract. No number recorded.quantization/quip-sharp
verifiedmedium · 0.75
SGLang hardware-dependent kernel selection logic for FP4/FP8 GEMM
varies by backend - W4A16 (Marlin), W4A4 (NVFP4), W8A8 (FP8)quantization/qhw-sglang-kernel-selection
verifiedhigh · 0.85
ShadowKV (low-rank key cache on GPU, value cache offloaded)
none by quantization; keys are compressed by a LOW-RANK representation and values are offloaded to CPU rather than reduced in precision.From the abstract: supports up to 6x larger batch sizes and boosts throughput by up to 3.04x on an A100 GPU without sacrificing accuracy, evaluated on RULER, LongBench and Needle-In-A-Haystack across Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M…quantization/shadowkv
verifiedmedium · 0.78
SignRound (signed gradient descent for weight rounding optimization)
w4a16 (2-4 bits per weight, with optimized rounding)From the paper abstract: 'SignRound achieved absolute average accuracy improvements ranging from 6.91% to 33.22% at 2 bits, as measured by the average zero-shot accuracy across 11 tasks.' This is a large improvement over baseline RTN at 2-b…quantization/qhw-signround
verifiedhigh · 0.85
Silent precision degradation: when W4A4 formats are served as W4A16 on pre-Blackwell hardware
w4a4 advertised, w4a16 actual on pre-Blackwell hardwarequantization/qhw-silent-precision-degradation
verifiedhigh · 0.90
SmoothQuant (W8A8 via offline outlier migration)
w8a8From the paper, OPT-175B under INT8 with the most aggressive O3 setting: 7 zero-shot benchmarks plus WikiText perplexity, reported as maintaining the FP16 model's accuracy (quant-smoothquant-paper); up to 1.56x speedup and 2x memory reducti…quantization/smoothquant
verifiedmedium · 0.80
SnapKV (observation-window KV selection) - NOT KV quantization
none - selects important KV POSITIONS per attention head from an observation window; no bits are reduced.From the abstract: 3.6x increase in generation speed and 8.2x memory-efficiency improvement over baseline at 16K-token inputs, comparable performance across 16 long-sequence datasets, and only a negligible accuracy drop on Needle-in-a-Hayst…quantization/snapkv
verifiedmedium · 0.80
SpinQuant (learned rotations + W4A4KV4)
w4a4 (weights AND activations at 4-bit; also quantizes the KV cache to 4-bit)From the paper (quant-spinquant-paper): W4A4KV4 on LLaMA-2 7B gives an average zero-shot reasoning accuracy of 64.0, a 2.9-point gap from full precision; LLM-QAT is 22.0 points behind under identical precision, SmoothQuant 25.0 points behin…quantization/spinquant
verifiedmedium · 0.75
SpQR (Sparse-Quantized Representation for near-lossless 3-4 bit compression)
w4a16 (3-4 bits per weight with sparse outlier preservation)From the paper abstract: 'relative accuracy losses of less than 1% in perplexity for highly-accurate LLaMA and Falcon LLMs' at 3-4 bits per parameter. This is a stronger claim than GPTQ or AWQ make for the same bit width. The paper also rep…quantization/qhw-spqr
verifiedhigh · 0.85
SqueezeLLM (dense-and-sparse quantization, up to 3-bit)
3-bit weights (paper claims lossless compression to ultra-low precisions of up to 3-bit); activations fp16From the abstract: on LLaMA models, 3-bit quantization 'significantly reduces the perplexity gap from the FP16 baseline by up to 2.1x as compared to the state-of-the-art methods with the same memory' budget. The 2.1x is a RATIO OF THE GAP, …quantization/squeezellm
verifiedmedium · 0.80
TensorRT-LLM per-GPU-architecture quantization support matrix (which format has a kernel on which arch)
matrix record - spans NVFP4, MXFP4, FP8 (per-tensor / block-scaled / rowwise), FP8 KV, NVFP4 KV, AWQ and GPTQ at W4A8/W4A16quantization/qhw-trtllm-hardware-matrix
verifiedhigh · 0.95
Ternary / 1.58-bit quantization (general)
1.58 avg (weights in {-1, 0, +1})quantization/ternary
verifiedlow · 0.60
Transformers bitsandbytes integration (load_in_4bit / load_in_8bit via BitsAndBytesConfig)
4-bit weights stored as NF4 or FP4; nested/double quantization adds a second pass over the already-quantized weights to store their own scales, saving a further 0.4 bits/parameter. 8-bit path is LLM.int8()n/a - the transformers doc gives no controlled perplexity delta for the 4-bit path. Null + note.quantization/bitsandbytes-load-in-4bit
verifiedhigh · 0.90
vLLM Implementation x Hardware support matrix (which quantization implementation runs on which architecture, and where it does NOT)
matrix record - spans AWQ, GPTQ, Marlin, llm-compressor INT8 W8A8/W4A8, llm-compressor FP8 W8A8, bitsandbytes, DeepSpeedFP, GGUFquantization/qhw-vllm-supported-hardware-matrix
verifiedhigh · 0.92
vLLM per-format kernel capability matrix derived from source (which format has a NATIVE GEMM on which GPU generation, which silently degrades to emulation, and which is a verified negative)
matrix record - spans NVFP4 (W4A4 and W4A16), MXFP4, MXFP6, MXFP8, FP8 per-tensor, FP8 block-scaled, FP8 rowwise, INT8 W8A8, W4A8, AWQ/GPTQ W4A16, and the 4-bit/3-bit KV-cache dtypesquantization/qhw-vllm-kernel-capability-matrix
verifiedhigh · 0.90
vLLM quantization backend selection and hardware compatibility
varies by method - W4A16 (GPTQ/AWQ/Marlin), W8A8 (FP8/INT8), W4A4 (NVFP4 on Blackwell)quantization/qhw-vllm-kernel-selection
verifiedhigh · 0.85
W4A4 (4-bit weights and 4-bit activations, general term)
w4a4quantization/wq4a4
verifiedmedium · 0.80
ZeroQuant v1 (fine-grained hardware-friendly PTQ + layer-by-layer distillation)
w8a8 headline; also int4 weights in fully-connected modules with int8 weights in attention and int8 activations (a mixed w4a8 configuration)From the paper's abstract: INT8 weight+activation is cost-free for BERT and GPT-3 style models with minimal accuracy impact, giving up to 5.19x (BERT) and 4.16x (GPT-3 style) speedup over FP16; on GPT-J 6B and GPT-NeoX 20B the INT8 model ac…quantization/zeroquant-v1
verifiedmedium · 0.75
ZeroQuant-V2 (systematic PTQ study + low-rank compensation LoRC)
study spans w4a16 and w4a8; also covers weight-only and activation-only variants across 125M-176B modelsNo single number in the abstract. The finding itself is the result: activation quantization is more susceptible to damage than weight quantization, and smaller models often tolerate activation quantization better than larger ones.quantization/zeroquant-v2
verifiedmedium · 0.75
ZipCache (channel-separable tokenwise KV quantization + salient token identification)
mixed precision across salient and non-salient tokens; headline is a 4.98x KV compression ratio, not a single bit widthFrom the abstract: on Mistral-7B / GSM8k, compresses the KV cache by 4.98x with only a 0.38% drop in accuracy. On LLaMA3-8B at input length 4096: 37.3% reduction in prefill-phase latency, 56.9% reduction in decoding-phase latency, 19.8% red…quantization/zipcache
verifiedmedium · 0.80
GGUF Q4_2 / Q4_3 (REMOVED types, type ids 4 and 5 reserved as DEPRECATED)
n/a - removed. The enum slots survive as .type_name = "DEPRECATED" with blck_size = 0, type_size = 0, is_quantized = falsen/a - no quality figure is published for a removed type; recorded as deprecated rather than given a fabricated delta. The live legacy figures are in the sibling records (gguf-q4-0, gguf-legacy-q4-1).quantization/gguf-deprecated-q4-2-q4-3
deprecatedhigh · 0.95
Most referenced
The records other records point at. Useful as an entry point: a heavily referenced accelerator or engine is one many decisions in this repo assume.