23 — Choosing a quantization format for a given hardware target

The decision this document serves: you have a hardware target and a workload, and you must pick a quantization format for it. The answer is not "the best format" — the repo has no defensible ranking, and §9 says why. It is a match between three things that are recorded separately and rarely read together:

the format  ×  the kernel that will actually run it  ×  the engine that chooses the kernel

Any two of the three being right is not enough. Most of the expensive failures in this repo are a format record and a kernel matrix agreeing with each other while the running server does something else.

For why weight-only 4-bit and W4A4 are different regimes, read 04-quant-selection.md first; this doc assumes that distinction.

0. The decision rule, short form

1. Rule 1 — a format record tells you what exists, not what runs

The single most common expensive mistake is reading native_support on a format record as a statement about your GPU. It is a statement about the format. Two records in this repo exist purely to make that distinction, and they are the most decision-relevant material here:

• TensorRT-LLM per-GPU-architecture quantization support matrix (which format has a kernel on which arch) — TensorRT-LLM's own per-architecture kernel table, eleven format columns from NVFP4 to W4A16 GPTQ, quoted row by row.
• vLLM Implementation x Hardware support matrix (which quantization implementation runs on which architecture, and where it does NOT) — vLLM's Implementation × Hardware table, nine implementation rows across Volta→Hopper plus AMD, Intel, x86 CPU and Arm CPU.

Both are filed under a proxy scheme value (nvfp4 and gptq respectively) because the schema's enum has no matrix member, and both say so in their own notes: a query for NVFP4 returns the matrix, not the format. Read the format record for the format and the matrix for the kernel; never quote one as the other. The TRT-LLM matrix says it explicitly: "Read this table as 'what TensorRT-LLM can do', never as 'what this GPU can do'."

1.1 The sm120 inversion, which is the row that changes a purchase decision

The TRT-LLM matrix row for Blackwell sm120 — the RTX 50-series and RTX PRO 6000 — reads: NVFP4 Y, MXFP4 Y, FP8(per tensor) Y, FP8(block scaling) ., FP8(rowwise) ., FP8 KV Y, NVFP4 KV ., and all four AWQ/GPTQ cells ..

That last clause is the finding. On Ampere the same four cells are: W4A8 ., W4A16 Y, W4A8 ., W4A16 Y. On Ada and Hopper all four are Y. So the consumer Blackwell parts invert the Ampere/Ada pattern at the bottom of the precision range while gaining formats at the top — the one generation in the table where the newest 4-bit format is available and the most popular 4-bit format is not, in the same engine (TensorRT-LLM per-GPU-architecture quantization support matrix (which format has a kernel on which arch)).

Three consequences to carry:

1. "My 5090 runs NVFP4" and "my 5090 runs AWQ" are both true, in different engines. TRT-LLM has the NVFP4 kernel and no AWQ kernel; vLLM and SGLang serve AWQ W4A16 on Blackwell through Marlin and Machete, which is a different kernel set entirely. The matrix record carries the caveat verbatim.
2. sm120 has no block-scaled FP8 GEMM and no NVFP4 KV-cache path, both of which sm100/103 and Rubin carry. Two of the three "Blackwell supports it" reflexes are wrong on the consumer part.
3. sm120 is also the row that the kernel records disagree about (Consumer Blackwell (SM120) vs datacenter Blackwell (SM100a): one architecture name, two incompatible kernel targets: sm100a-compiled kernels "are not compatible with RTX 50 series GPUs"; FlashInfer: a kernel-selection layer over FlashAttention, cuDNN, CUTLASS and TensorRT-LLM notes FlashInfer lists SM100/SM103 and SM120/SM121 separately). §8 keeps this open.

1.2 FP8 rowwise is a Hopper-only format, which is the opposite of intuitive

FP8 rowwise scaling needs no per-tensor calibration, so it is the FP8 variant people reach for by default. In the vendor matrix it is Y on Hopper only — . on sm120, sm100/103, sm107, Ada and Ampere (TensorRT-LLM per-GPU-architecture quantization support matrix (which format has a kernel on which arch)). It is the narrowest-supported FP8 variant in the table despite having the easiest calibration story.

Do not treat "FP8" as one row. The matrix's own recipe line is arch-dependent: "FP8 block wise scaling GEMM kernels for sm100/103/107 are using MXFP8 recipe (E4M3 act/weight and UE8M0 act/weight scale), which is slightly different from SM90 FP8 recipe (E4M3 act/weight and FP32 act/weight scale)." One column covers two numerically different scale formats, so a memory or quality calculation for block-scaled FP8 that assumes the SM90 recipe is wrong on Blackwell in a way the table does not flag.

Independent corroboration from the other direction: SmoothQuant with FP8 activations: does outlier migration still help? notes "the FP8 format has limited precision (E4M3 has 3 mantissa bits), so the migration may not help as much as it does for INT8" — SmoothQuant was designed for INT8, and whether it still earns its keep against FP8 is recorded as an open research question, not a result.

1.3 The W4A8 row is a backend-selection fact, not a hardware fact

vLLM's llm-compressor INT8 W4A8 row is supported on Arm CPU and on nothing else in the matrix — not on any NVIDIA generation, not AMD, not Intel, not x86 (vLLM Implementation x Hardware support matrix (which quantization implementation runs on which architecture, and where it does NOT)). The obvious fallback "the GPU can't do W4A8, so put it on the CPU" is false on x86, and that is the shape of the trap: a . in an NVIDIA column reads like a hardware limit when the reason is that the implementation has one backend.

Three more cells in the same table that a flat reading gets wrong:

• GPTQ is the only row that covers Volta. AWQ and Marlin both exclude SM7.0. "GPTQ is the older format" is backwards as a compatibility statement: on a V100 it is the only one of the three that runs.
• Marlin's Turing cell carries a footnote narrower than the cell. The star expands to "Turing does not support Marlin MXFP4", so the cell means Marlin-minus-MXFP4; a reader treating the column as a flat yes picks MXFP4 on a T4 and fails.
• FP8 W8A8 is Ada-and-Hopper-only in this table, which is narrower than "FP8 works on Ampere and up" — on the Ampere row bitsandbytes and GGUF are YES while llm-compressor FP8 is NO, at the same compute capability.

The matrix covers Volta through Hopper only; Blackwell is absent from it. That is why it must be read together with the TRT-LLM matrix rather than instead of it, and it is why its own scope line warns the chart "is subject to change as vLLM continues to evolve".

2. Rule 2 — verify which kernel is actually running, every deploy

Given §1, the format you chose is a hypothesis. The running kernel is the fact. Four records describe the same failure from four directions, and it is the most consequential class in the repo: the output is correct and the performance is not what you bought.

• Silent precision degradation: when W4A4 formats are served as W4A16 on pre-Blackwell hardware (confidence 0.9) is the primary record. SGLang's --fp4-gemm-backend auto-selection order is "1) flashinfer_cutedsl on SM100,
  2. marlin on SM80-SM90, 3) flashinfer_cutlass otherwise (including SM120)". Marlin is a W4A16 kernel, so an NVFP4 checkpoint on an A100 or H100 is served with FP4 weights dequantized to FP16 and FP4 activations upcast to FP16. The record's own phrasing is "The user sees the memory saving but not the compute speedup", and the two kernel-selection records state the consequence as "The memory saving materializes but the compute speedup does not" (Silent precision degradation: when W4A4 formats are served as W4A16 on pre-Blackwell hardware, SGLang hardware-dependent kernel selection logic for FP4/FP8 GEMM, vLLM quantization backend selection and hardware compatibility). The same behaviour is documented for vLLM's modelopt_fp4: SGLang's platform table, quoted in its record, "shows that modelopt_fp4 uses 'Marlin W4A16 fallback on Ampere/Hopper and native FP4 backends on Blackwell'" (SGLang hardware-dependent kernel selection logic for FP4/FP8 GEMM).
• The framework names the condition. vLLM's --kernel-config example is --linear-backend cutlass --kernel-config {"linear_backend_per_quant":{"nvfp4_w4a16":"humming"}} — nvfp4_w4a16 is the scheme name a W4A16-served NVFP4 checkpoint carries in the config (vLLM quantization backend selection and hardware compatibility).
• And the roofline explains why it costs you. Decode-phase GEMM: memory-bandwidth bound at batch size 1: "at decode time with batch size 1, the GEMM is memory-bound, so reducing weight bytes directly increases tokens/s ... when NVFP4 is served with Marlin (W4A16) on pre-Blackwell hardware, the weight bytes are reduced but the activation bytes are not, so the speedup is less than the theoretical maximum."

Three checks that cost nothing and catch all of it:

The third check exists because the warning you would naturally trust is a measurement bug, not a capability statement: "Your GPU does not have native support for FP4 computation" is emitted for a weight-only checkpoint, and the record's instruction is "Do not read this warning as a hardware capability statement."

The inverse mistake is also recorded and it is worse, because it is silent and wrong in the other direction: FlashInfer's b12x MoE backend silently defaults quant_mode to W4A4, so a weight-only NVFP4 checkpoint is computed at the wrong activation precision (major) — a checkpoint marked W4A16_NVFP4 served with --moe-backend flashinfer_b12x runs the FP4×FP4 micro-kernel, and the record's instruction is "Do not accept a b12x trace showing an FP4 kernel for a W4A16 checkpoint." Two benchmark records price that misroute on an identical model: Qwen3-8B MXFP4 W4A16 served on MarlinMxfp4Kernel (correct W4A16 path): AIME-24/25 + GSM8K + MATH-500 average accuracy 78.16 versus Qwen3-8B MXFP4 W4A16 served on the FlashInfer W4A4 kernel: AIME-24/25 + GSM8K + MATH-500 average accuracy 70.58, against the unquantized Qwen3-8B BF16 reference: AIME-24/25 + GSM8K + MATH-500 average accuracy (baseline for the MXFP4 W4A16 dispatch comparison) 82.36. A 7.58-point accuracy drop caused purely by kernel selection.

3. Rule 3 — on Blackwell, NVFP4 beats MXFP4 on risk and loses on portability

Both are W4A4 with real FP4 tensor-core paths, and they are not interchangeable — a model quantized to MXFP4 cannot be served with NVFP4 kernels or vice versa (MXFP4 vs NVFP4: block size, scale format, and hardware support comparison).

Recommendation: NVFP4 by default, MXFP4 when portability is a requirement. The accuracy argument is the vendor's own: NVIDIA's comparison table rates MXFP4 as "risk of noticeable accuracy drop compared to FP8" where NVFP4 is "lower risk", and the quantitative reason is that encoding the scale in E4M3 rather than E8M0 "drops the scale-encoding MSE from 0.72 to 0.08" (NVFP4 (Blackwell block-scaled E2M1 + E4M3), MXFP4 vs NVFP4: block size, scale format, and hardware support comparison).

Two honesty constraints on that recommendation. The MSE ranking is a vendor claim, and MXFP4 vs NVFP4: block size, scale format, and hardware support comparison records that "no independent controlled benchmark comparing MXFP4 and NVFP4 on the same model with the same calibration data was found." And MXFP4 (OCP microscaling FP4, E8M0 scale) has no inference quality delta recorded at all — the closest evidence is a PTQ study of SmoothQuant+AWQ+GPTQ on MX formats, which is W4A8, not W4A4. So the format preference rests on a vendor table, and the MXFP4 record says in its own words that no MXFP4 accuracy number should be quoted from it.

Both formats are Blackwell-only in silicon; on Hopper and Ampere both are emulated_support, which is §2's failure mode rather than a slow path you can accept.

The mechanism underneath is in Block-scaled FP8/FP4 tensor cores: MXFP and NVFP formats inside the MMA instruction: the scale is "part of the MMA OPERAND, not a separate kernel", which is why "the activation quantization no longer needs a fused dequant pass". Its portability warning is the one to remember when you change parts: "SM90 requires scaling factors in FP32 while SM100 requires them packed in UE8M0 ... So switching an FP8 model from H100 to B200 changes the memory layout the engine must produce, not just the kernel it calls."

MoE is a separate question from dense. MXFP4 MoE quantization: per-expert scales and block scaling for mixture-of-experts (draft, 0.5) records that per-tensor rather than per-expert scales are used for MoE, that "the block scaling in MXFP4 (32 elements) may not align well with MoE's expert structure", and flags "a potential source of silent accuracy degradation that has not been well-documented." Treat MoE MXFP4 as unproven on accuracy; do not extrapolate the dense result.

4. Rule 4 — W4A16: group size is the scheme, and the kernel is chosen for you

4.1 Group size 128 is not a tunable

AWQ (activation-aware weight quantization): "Group size 128 is the scheme's defining choice and the file layout - it is not a tunable." GPTQ (one-shot approximate second-order PTQ) adds that 32/64/1024 "are legal and change both quality and kernel eligibility, which is a frequent source of 'GPTQ is slow' reports." If you are handed a GPTQ file at group 64, you are not holding the format the docs describe, and the kernel eligibility is a separate question from the quality.

The 128 default is also the recommended default in the toolkit that implements the low-bit path: AutoRound confirms "bits 2 through 8, every one with group_size=128 and sym=True and act_bits=16", and its troubleshooting section recommends 32 or mixed bits for better results — "so 128 is the default, not the optimum" (AutoRound (Intel) - SignRound weight-rounding PTQ toolkit, W4A16 and neighbours).

The asymmetry limit is the practical line in that record. --asym supports weight bits ≤ 7 for the auto_round/auto_gptq/auto_awq exports, "because vLLM serves W8 GPTQ-format weights symmetric-only and Marlin supports zero points at 4 bits only." Two constraints, one from each side of the stack, that agree.

And a labelling trap in the same record: its README labels MXFP4, MXINT4, MXFP4_RCEIL, MXFP8, FPW8A16 and FP8_STATIC as "Research feature, no real kernel" — do not read them as deployable.

4.2 Which kernel runs, per generation — three records, one precision

Marlin INT4 weight-only kernel (FP16xINT4 mixed-precision GEMM) and Machete Hopper-optimized mixed-precision GEMM kernel (CUTLASS-based) are the decision records, and both carry the same load-bearing fact: Marlin and Machete produce the same numerical results. Marlin's is explicit — "despite being called an 'INT4 kernel', the actual tensor-core operation is FP16 × FP16 - weights are dequantized to FP16 in registers before the multiply. This is NOT an INT4 tensor-core path (which does not exist on Ampere/Ada). The speedup comes from reduced memory bandwidth (4-bit weight storage), not from faster arithmetic." Machete's record says the same for Hopper: "the kernel is numerically equivalent to Marlin for the same quantized weights, so the Marlin quality figures apply."

So the kernel choice is purely a performance decision, and it is invisible in the output. Selection logic, from Machete Hopper-optimized mixed-precision GEMM kernel (CUTLASS-based): "vLLM uses Machete when the GPU is SM90 (Hopper) and the quantization method is AWQ or GPTQ. On Ampere/Ada, Marlin is used instead."

The honest comparison is in the MMQ record and it should be quoted whenever anyone ranks these: llama.cpp "reaches 4-bit throughput through integer SIMT dot products with a broad arch reach (including RDNA and CDNA), while Marlin and Machete reach it through tensor cores on a narrower arch list. Neither is strictly better ... and any comparison must state which path was taken."

Two more kernel facts that cost people deployments. Marlin is a GEMM kernel, not a format, and both records say so — Marlin "is filed under 'gptq' because Marlin consumes GPTQ-packed weights and the enum has no kernel-family values", and the proxy will "make Machete under 'awq' and Marlin under 'gptq' look like two different precisions when they are the same one". And Marlin's ROCm path has a hard failure worth pre-checking: The ROCm Triton w4a16 path asserts on GPTQ models with a non-zero-group qzeros layout (major), a qzeros.shape == (K // group_size, N // 8) assertion when serving GPTQ-Int4 on the ROCm Triton w4a16 path.

4.3 The accuracy budget you can actually cite

Accuracy cost of quantization: measured deltas on named benchmarks compiles the deltas that papers actually report, and its pattern line is the most quotable summary in the repo: "4-bit weight-only quantization typically costs 0.1-0.3 ppl on WikiText-2. 3-bit costs 0.5-1.0 ppl. 2-bit costs 2.0+ ppl. KV cache quantization at 4-bit is nearly lossless; at 2-3-bit it costs 0.1-0.5 ppl." The same record's own caveat is the load-bearing part: "The accuracy cost is model-dependent and benchmark-dependent."

The controlled numbers behind it, with models attached:

Two rows in that table are the actionable ones. At 3 bits, group size buys more than the method does (+0.93 vs +2.39 ppl for the same format at the same bit width). And rounding is a first-class lever at 2 bits, where SignRound (signed gradient descent for weight rounding optimization) records "6.91-33.22% absolute average accuracy improvement" — but it is a rounding algorithm, it "cannot be combined with other quantization methods (like AWQ's scaling or GPTQ's Hessian-based rounding) without re-running the optimization," and it is implemented by AutoRound (AutoRound (Intel) - SignRound weight-rounding PTQ toolkit, W4A16 and neighbours).

4.4 Bitrate is a target, not a format — EXL2/EXL3

EXL2 (GPTQ with mixed per-tensor bitrate) mixes quantization levels within a model, and can "apply multiple quantization levels to each individual linear layer - to hit any target average bitrate between 2 and 8 bpw." Its successor EXL3 (ExLlamaV3 format, QTIP-based trellis quantization) - the successor to the archived EXL2 is a different format with "no checkpoint compatibility between them" (ExLlamaV3) — a lattice-coded, Hessian-calibrated QTIP derivative rather than a grid quantizer.

The measured throughput curve is the decision-relevant part, from ExLlamaV2 EXL2 kernel throughput benchmarks (3090 Ti / 4090): lower bit width is slower, not faster. 2.5 bpw 70B runs 33/38 t/s on a 3090 Ti / 4090 while 4.0 bpw 7B runs 185/211 — because "lower bit widths require more complex dequantization and the memory bandwidth saving is offset by increased compute overhead." The same mechanism is QServe / QoQ (W4A8KV4 quantization and system co-design)'s central claim from the serving side: "existing INT4 methods suffer 20-90% runtime overhead from dequantizing weights or partial sums on GPUs", and INT4 "can be slower than fp16" unless you also fix the dequant path.

Neither record carries a quality delta — EXL2 (GPTQ with mixed per-tensor bitrate)'s quality_delta is empty because the README publishes throughput only, and ExLlamaV2 EXL2 kernel throughput benchmarks (3090 Ti / 4090) says "No quality delta is published in the README - only throughput numbers." Choosing EXL2/EXL3 is choosing an unquantified quality risk for a bitrate target. EXL3's group-size 256 is also the embedding group specifically — "Each embedding is split into 256-wide groups that are rotated and trellis-coded without calibration, and the following layers are calibrated against the quantized table" — which is §5's trap in a second disguise.

5. Rule 5 — GGUF: match bits-per-weight, and do not trust a group size of 256

5.1 The group-size correction, which changes what you can compare

Eight GGUF records previously carried weight_group_size: 256. Seven of those were wrong, and the error was the K-quant super-block size being read as the scale group. The correction was made from the ggml dequantize routines, and it matters because 256 vs 16/32 is the difference between "one fp16 scale per 256 values" and "eight or sixteen sub-block scales per 256 values".

Four records are legitimately at 256 and must not be "corrected" again, because for them the super-block and the scale group genuinely coincide: GGUF IQ1_S (1.5625 bpw, non-linear 1-bit-ish with a 16-bit grid high plane) ("256 IS right here and the distinction matters" — one fp16 d for 256 values, no sub-block scales), GGUF IQ2_XXS (2.0625 bpw, almost-true 2-bit with a 16-bit scale per 256), GGUF IQ3_XXS (3.0625 bpw, almost-true 3-bit with a 16-bit scale per 256), and GGUF TQ1_0 / TQ2_0 (ternarized quants, QK_K super-block, trit-packed) ("TQ1_0 genuinely has a 256-value scale group ... unlike the K-quants").

The rule this yields: in GGUF the number that matters for storage is bits-per-weight, and the number that matters for quality comparison is bits-per-weight on a matched budget — not a group size, and never a quant name. See §5.2.

5.2 Measured bpw, the number to match on

Llama-3.1-8B, measured average bits per weight, straight from the records:

Five K-quant records are contested because the measured bpw of the *_M mixture types does not match the pure-block struct comments; both numbers are retained per record.

Two names are not two configurations. GGUF Q4_K_M (4-bit K-quant, medium): "'Q4_K_M IS A MIXTURE, not a type'" — base q4_K with attn_v and ffn_down promoted to Q6_K, and "The plain 'Q4_K' CLI name is an alias for Q4_K_M". The consequence is that a quant name does not identify a bit budget across models; see Two models quantized at 'the same Q4_K_M level' can land at 4.75 and 4.30 bpw, so named-quant comparisons across models are not controlled experiments and llama.cpp's --target-bpw, which is the tool that fixes it.

The GGUF case of the mixture trap is bigger than the K-quant one. GGUF MXFP4_MOE (llama.cpp MoE-specific ftype: MXFP4 for experts, Q8_0 for everything else) is llama.cpp's LLAMA_FTYPE_MOSTLY_MXFP4_MOE: MXFP4 on expert tensors and Q8_0 on everything else — "the embedding, output head, and any non-expert projection are stored at 8.5 bpw while the experts sit at 4.25. So the whole-model bpw of an MXFP4_MOE model is NOT 4.25**." Its mapping is explicit about 3-D tensors and BailingMoE3 MLA projections. Note also that for MX formats "the scale group IS the block" (QK_MXFP4 32), which is why GGUF MXFP4_MOE (llama.cpp MoE-specific ftype: MXFP4 for experts, Q8_0 for everything else) and MXFP4 (OCP microscaling FP4, E8M0 scale) agree at 32 and need no correction.

5.3 The lever that costs nothing: imatrix

GGUF imatrix (importance matrix) - the calibration file that determines WHICH weights get more bits is "not a format - a per-tensor activation-energy file" whose own note calls it "THE MAJOR PRACTICAL LEVER NOBODY RECORDS". It applies across every format ("every ggml type, every bit width, legacy and k-quant and i-quant alike") and its effect "is to REMOVE part of the quantization delta of the format it is applied to", with official per-type figures Q4_K_M +0.1754 and Q5_K_M +0.056 — which is most of GGUF Q4_K_M (4-bit K-quant, medium)'s +0.24 total delta.

If you are quantizing a GGUF yourself, this is a higher-return action than choosing a different quant type, and it composes with nothing else in this document.

5.4 GGUF has no silicon anywhere

Worth stating because it inverts the usual question. GGUF IQ4_XS (non-linear 4-bit, 256-value super-block with 6-bit sub-block scales): "so native_support is deliberately EMPTY and the whole accelerator ecosystem is emulated_support. No accelerator has an i-quant or k-quant path in silicon. This is the single most commonly botched field in this repo." No GGUF quant will ever be faster because the GPU supports it; it is faster because there are fewer bytes, reached by MMQ (llama.cpp MMQ: INT8 activation quantization with per-arch tile configs, no tensor cores).

6. Rule 6 — 8-bit KV cache: widely deployed, and completely unmeasured

This is the recommendation with the weakest evidence base in the repo, and the honest thing is to say so rather than pick a winner.

FP8 KV cache quantization (E4M3 keys and values) and INT8 KV cache quantization (8-bit keys and values) are both draft at confidence 0.5, and both quality_delta fields say the same thing: no controlled benchmark with a named accuracy delta was found. FP8 KV's field in full: "No controlled benchmark with named accuracy delta found in the sources accessed ... This is a significant gap: FP8 KV cache is widely deployed but the accuracy impact is not well-documented in public sources."

What is recorded, and it is enough to act on:

• Both halve KV memory versus FP16.
• The kernel interaction is not free: "the KV cache is quantized after the KV projection, so the attention kernel must dequantize before computing attention scores. This dequantization adds latency to the decode phase."
• Support is not uniform: FP8 KV is native on Hopper and Ada, "On Ampere (SM80), FP8 is emulated via FP16 with reduced precision", and "The accuracy delta is expected to be larger on Ampere where the FP8 path is emulated."
• INT8's record argues a mechanism preference: "INT8 is symmetric around zero while FP8 has an exponent. For attention keys that feed into softmax, the INT8 format may be more appropriate because the key distribution is roughly symmetric." That is a may in the record's own words.
• And a silent-corruption combination exists, which outranks the accuracy question: Disabling FlashInfer Q-quantization with an FP8 KV cache and speculative decoding silently corrupts sliding-window models on SM100 — FP8 KV plus sliding-window attention plus speculative decoding, which is the exact configuration the FP8 KV record itself warns about.

Recommendation: use 8-bit KV to fit the cache, not to save accuracy risk — you are accepting an unquantified delta in exchange for 2x capacity — and never combine it with sliding-window attention and speculative decoding until you have measured that combination. Establish the delta yourself; the repo cannot do it for you.

For 4-bit and below, there is no such ambiguity, only disagreement. The family disagrees about the right key bit width and the repo says so: "Note the whole family (KIVI 2/4, KVQuant 3/3, QuaRot 4/4, QServe 4/4, ZipCache mixed) disagrees about the right key bit width - there is no settled answer" (QServe SmoothAttention (4-bit KV cache with an accuracy-preserving correction)).

Recommendation: KVQuant 3-bit if you want the best-documented number; KIVI if you want the most capacity per unit of risk, accepting a qualitative claim; QServe 4-bit only with SmoothAttention attached, since its record is explicit that SmoothAttention "is a CORRECTION METHOD ... introduced specifically to counter the accuracy degradation that 4-bit KV quantization causes." And note the asymmetry result is KIVI's central finding, not an incidental detail — do not carry 4-bit KV and 4-bit values forward to 2-bit keys assuming the halves are independent.

One non-quantization answer is sometimes correct. Pensieve (multi-tier conversation-state KV cache) - NOT KV quantization "solves the SAME problem as KIVI ... by storing it across tiers instead of compressing it", and the record's judgement is the useful part: "Since lossless storage is strictly better than lossy compression when the constraint is host memory rather than GPU memory, Pensieve is the correct answer when an offload path exists and KIVI is the correct answer when it does not."

7. Rule 7 — the non-post-training families, which are not choices at all

Two families in the records look like format choices and are not, and picking them as if they were is a category error.

• Ternary / BitNet b1.58 — Ternary / 1.58-bit quantization (general), BitNet b1.58 (ternary LLM): "ternary LLM quantization is a TRAINING-TIME recipe, not a post-training one. There is no such thing as PTQ-ing an existing fp16 LLM to 1.58 bits and keeping its quality." The paper's "matches fp16" is a statement about the ternary set's expressiveness under an identical token budget. native_support is empty on both records. You cannot select this for an existing checkpoint.
• W4A4 research without silicon — SpinQuant (learned rotations + W4A4KV4) is genuinely W4A4 (64.0 average zero-shot on LLaMA-2 7B, 2.9 points from full precision, SpinQuant W4A4KV4 8-task zero-shot accuracy delta vs fp16, LLaMA-2 7B) but "there is NO hardware INT4 tensor-core path. Its W4A4 numbers come from research Triton kernels plus fake quantization". Atom (mixed-precision block-outlier low-bit quantization for serving) is W4A4 with real integer operators, and its record draws the distinction that matters: Atom "is built around 4-bit integer operators - but that is an integer-operator path on datacenter GPUs, not the FP4 tensor-core path that Blackwell introduced". Cite these for accuracy evidence; do not cite them for speed. Its own throughput claim (7.73x vs FP16, 2.53x vs INT8) is from the authors' own kernels.

8. What would change these recommendations

9. What the records cannot answer

1. There is no cost-per-quality ranking of formats. The deltas in §4.3 are isolated perplexity differences at isolated bit widths with no shared baseline methodology, and every one of them has its model name attached. Two of the cleanest comparisons are within a family (Marlin vs AWQ on the same model; NVFP4 vs MXFP4 on the same card) and neither crosses a bit width.

2. The vLLM matrix stops at Hopper and the TRT-LLM matrix stops at Rubin. Blackwell's vLLM story has to be inferred from the other table plus the sm120 kernel records, which disagree with each other (Consumer Blackwell (SM120) vs datacenter Blackwell (SM100a): one architecture name, two incompatible kernel targets vs the format claim that sm120 is native for NVFP4). No record reconciles them, and there are no native_support entries for the RTX 5090 or RTX PRO 6000 from any format record — they appear in matrix records and emulated_support lists only.

3. Zero controlled accuracy data for 8-bit KV cache, and the least-documented area named by the records themselves is "FP8/FP4 KV cache quantization, where no controlled benchmark with named accuracy delta was found" (Accuracy cost of quantization: measured deltas on named benchmarks).

4. Format quality deltas are overwhelmingly Llama-3.1-8B or Llama-2-7B. No format has a controlled delta on a modern MoE, which is exactly where quantization behaviour is hardest — and MXFP4-MoE is the place where the format and the model structure are recorded as possibly mismatched. The sub-1-bit exception, QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models, reports a masked-LM validation loss on SwitchTransformers (c2048 ternary 1.26 against a bfloat16 1.18) rather than a task score, and its kernel only wins as a batch-1 GEMV (The QMoE sub-1-bit expert kernel is a GEMV, and it loses as soon as two tokens share an expert).

5. The W4A4 speed claim on Blackwell is vendor-only. The 3.5x memory and 25x/50x energy figures in NVFP4 (Blackwell block-scaled E2M1 + E4M3) are labelled "marketing, not measurements" by the record itself.

6. EXL2 and EXL3 have no quality delta at all, and EXL2 (GPTQ with mixed per-tensor bitrate) is draft at 0.6 against an archived upstream (ExLlamaV2).

7. SmoothQuant-on-FP8 is a research question, not a result (SmoothQuant with FP8 activations: does outlier migration still help?, draft, confidence 0.4 — the lowest-confidence record cited here).

8. AMD has no vLLM AWQ/GPTQ path and no Quark record in this set. The vLLM matrix's AMD column is a whole-column NO that "reads like a data gap" and "must not be read as 'AMD GPUs cannot do quantized inference'" — the AMD-native route the record names is covered by quark and rocm-quant records that this document does not cite.

9. A schema gap that makes Apple advice indirect, and a 04/23 consistency note. MLX affine quantized linear (mlx-lm's default 4-bit weight format, group_size 64) is the native format on Apple records and lists apple-m3-max and apple-m3-ultra in native_support, at group 64 with emulated_support empty. 04-quant-selection.md §3.2 said "no native quantization support recorded at all" for those two parts; that text has been corrected to agree with this document, so the two no longer contradict each other and the base layer and the selection layer say the same thing about Apple. What remains open is not the native/emulated question but everything downstream of it:

   • MLX's own mlx-quantized-4bit / mlx-quantized-8bit engine labels still have no record of their own. The MLX affine record is the closest thing, and it is keyed on the affine mode with group 64 rather than on the engine's label.
   • Three benchmark records still carry format_id: mlx-quantized-4bit, which is not a record id — the format record's own notes name these as the dangling references its existence was meant to close, and repointing them is a data/benchmarks/ edit this wave does not own.
   • "Native" is not "comparable". MLX ships four modes at four group sizes (affine 64, mxfp4 32, nvfp4 16, mxfp8 32), so a checkpoint recorded as "4-bit on Apple" may be a 64-group affine grid or a 32-group MX grid, and those differ in memory arithmetic and quality. The nvfp4 mode at 16 matches the Blackwell convention in NVFP4 (Blackwell block-scaled E2M1 + E4M3) and the mxfp4 mode at 32 matches MXFP4 (OCP microscaling FP4, E8M0 scale); affine 64 matches neither.

   For the selection consequence on Apple: MLX affine 4-bit, group 64, and treat the group size as a default rather than a constant — mlx.nn.quantize takes group_size per call, so the number in the record is the API's default and not a property of the format.

Related documents

• 04-quant-selection.md — the regime distinction (W4A16 vs W4A4) this table assumes.
• 04-quant-selection.md — the regime taxonomy (W4A16 vs W4A4), the per-accelerator native/emulated census, and the quality tables this document draws on. Read §3 of that doc as the base layer; read this one for the selection step.
• 01-hardware-selection.md — which part to buy, per precision regime. The matrix rows here are the reason its Blackwell and Ampere recommendations differ.
• 03-engine-selection.md — which engine supports which format on which backend, and the build costs of the two engines named in §1.
• 05-known-traps.md — the full severity-ordered list behind §2.
• 09-model-memory-envelope.md — whether the format you picked actually fits, and what KV cache it leaves room for.
• 08-cost-per-token.md — what the precision choice costs per million output tokens, joined against real prices.