inference/research
verifiedhigh · 0.85updated 2026-10-08

Qwen3-8B MXFP4 W4A16 served on MarlinMxfp4Kernel (correct W4A16 path): AIME-24/25 + GSM8K + MATH-500 average accuracy

78.16 accuracy_percent (unweighted average of AIME-24, AIME-25, GSM8K, MATH-500)

Fields

id
benchmarks/mxfp4-w4a16-on-w4a16-marlin-aime-math-gsm8k-avg
Metric
quality
Value
78.16
Unit
accuracy_percent (unweighted average of AIME-24, AIME-25, GSM8K, MATH-500)
Model
Qwen3-8B MXFP4 W4A16 (compressed-tensors, MXFP4 weights group_size=32, input_activations: null)
Methodology
Same checkpoint, same harness, same prompts and sampling as mxfp4-w4a16-on-w4a4-kernel-aime-math-gsm8k-avg, with the only change being a manual switch to MarlinMxfp4Kernel - the W4A16 path the checkpoint's contract calls for. Qwen3-8B MXFP4 W4A16 on 8x B200 (SM100) with vLLM --dtype bfloat16 --tensor-parallel-size 1. Compared against the BF16 baseline (82.36 avg) and the mis-dispatched W4A4 kernel (70.58 avg).
Measured by
third_party
Reproducible
yes
notes
The corrected number for the checkpoint, and the one to compare any MXFP4 claim against. 78.16 vs BF16 82.36 = 4.20 pp for MXFP4 weight-only quantization on these four sets, which is the plausible cost; the further 7.58 pp in the sibling record is kernel mis-dispatch, not format cost. Component scores: AIME-24 66.67, AIME-25 55.83, GSM8K 94.62, MATH-500 95.53. Reporter notes the recovered result is 'basically consistent with the results reported in the public paper' - which is the check that confirms the diagnosis, since the paper measured W4A16. See gotcha mxfp4-w4a16-misrouted-to-w4a4-kernel. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-b200). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface. PROVENANCE RAISED 2026-10-08 (w8-bench-src): added [[w8-src-arxiv-2505-09388-qwen3-technical-report]], the authors' own Qwen3 Technical Report, fetched and read. It is the Tier-A document for the REFERENCE POINT of this comparison (its Table 17 gives Qwen3-8B thinking-mode MATH-500 97.4, AIME'24 76.0, AIME'25 67.3, GPQA-Diamond 62.0) and the reporter's own note that the recovered 78.16 is 'basically consistent with the results reported in the public paper' is now checkable against that paper rather than being an unsourced assertion in an issue. VALUE UNCHANGED. The paper's harness is not vLLM's and the paper does not average these four sets, so 78.16 remains a third-party measurement on a third-party harness - what changed is that its baseline now has a primary citation.

Notes

The corrected number for the checkpoint, and the one to compare any MXFP4 claim against. 78.16 vs BF16 82.36 = 4.20 pp for MXFP4 weight-only quantization on these four sets, which is the plausible cost; the further 7.58 pp in the sibling record is kernel mis-dispatch, not format cost. Component scores: AIME-24 66.67, AIME-25 55.83, GSM8K 94.62, MATH-500 95.53. Reporter notes the recovered result is 'basically consistent with the results reported in the public paper' - which is the check that confirms the diagnosis, since the paper measured W4A16. See gotcha mxfp4-w4a16-misrouted-to-w4a4-kernel. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-b200). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

PROVENANCE RAISED 2026-10-08 (w8-bench-src): added [[w8-src-arxiv-2505-09388-qwen3-technical-report]], the authors' own Qwen3 Technical Report, fetched and read. It is the Tier-A document for the REFERENCE POINT of this comparison (its Table 17 gives Qwen3-8B thinking-mode MATH-500 97.4, AIME'24 76.0, AIME'25 67.3, GPQA-Diamond 62.0) and the reporter's own note that the recovered 78.16 is 'basically consistent with the results reported in the public paper' is now checkable against that paper rather than being an unsourced assertion in an issue. VALUE UNCHANGED. The paper's harness is not vLLM's and the paper does not average these four sets, so 78.16 remains a third-party measurement on a third-party harness - what changed is that its baseline now has a primary citation.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (2)