inference/research
verifiedhigh · 0.85updated 2026-10-03

Qwen3-8B MXFP4 W4A16 served on the FlashInfer W4A4 kernel: AIME-24/25 + GSM8K + MATH-500 average accuracy

70.58 accuracy_percent (unweighted average of AIME-24, AIME-25, GSM8K, MATH-500)

Fields

id
benchmarks/mxfp4-w4a16-on-w4a4-kernel-aime-math-gsm8k-avg
Metric
quality
Value
70.58
Unit
accuracy_percent (unweighted average of AIME-24, AIME-25, GSM8K, MATH-500)
Model
Qwen3-8B MXFP4 W4A16 (compressed-tensors, MXFP4 weights group_size=32, input_activations: null)
Methodology
Qwen3-8B MXFP4 W4A16 checkpoint, vLLM --dtype bfloat16 --tensor-parallel-size 1 on 8x B200 (SM100). The checkpoint declares input_activations: null, i.e. no activation quantization, but vLLM routed it to FlashInferMxfp4LinearKernel (the W4A4 path) - visible directly in the startup log line 'Using FlashInferMxfp4LinearKernel for MXFP4 GEMM'. Evaluated on AIME-24, AIME-25, GSM8K and MATH-500 with the same system prompts, sampling configuration and evaluation harness for all three rows of the comparison (BF16 baseline 82.36 avg, MarlinMxfp4Kernel 78.16 avg).
Measured by
third_party
Reproducible
yes
notes
THIS IS THE BROKEN NUMBER, not the quantized model's true quality. The same weights on MarlinMxfp4Kernel score 78.16 avg, and BF16 scores 82.36. The 7.58 pp gap to Marlin is attributable to executing a W4A16 checkpoint on a W4A4 kernel; the remaining 4.20 pp is normal MXFP4 quantization loss. Component scores here: AIME-24 51.25, AIME-25 42.08, GSM8K 93.86, MATH-500 95.13. Reads as a plausible 'MXFP4 costs ~12 points' result and invites exactly the wrong conclusion. The tell is the startup log naming the FlashInfer MXFP4 kernel on a checkpoint whose config has input_activations null. See gotcha mxfp4-w4a16-misrouted-to-w4a4-kernel; vLLM PR #56803 fixes the dispatch. Sibling records cover the same checkpoint under mxfp4-w4a16-on-w4a16-marlin and the bf16 reference. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-b200). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

THIS IS THE BROKEN NUMBER, not the quantized model's true quality. The same weights on MarlinMxfp4Kernel score 78.16 avg, and BF16 scores 82.36. The 7.58 pp gap to Marlin is attributable to executing a W4A16 checkpoint on a W4A4 kernel; the remaining 4.20 pp is normal MXFP4 quantization loss. Component scores here: AIME-24 51.25, AIME-25 42.08, GSM8K 93.86, MATH-500 95.13. Reads as a plausible 'MXFP4 costs ~12 points' result and invites exactly the wrong conclusion. The tell is the startup log naming the FlashInfer MXFP4 kernel on a checkpoint whose config has input_activations null. See gotcha mxfp4-w4a16-misrouted-to-w4a4-kernel; vLLM PR #56803 fixes the dispatch. Sibling records cover the same checkpoint under mxfp4-w4a16-on-w4a16-marlin and the bf16 reference. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-b200). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (2)