inference/research
verifiedmedium · 0.80updated 2026-10-03

Qwen3.6-35B-A3B NVFP4 W4A16 MoE on the FlashInfer b12x backend: GSM8K accuracy (W4A16 checkpoint executed as W4A4)

0.9037 accuracy (GSM8K fraction correct, 5-shot as configured by the reporter's shared eval settings)

Fields

id
benchmarks/nvfp4-w4a16-flashinfer-b12x-moe-gsm8k
Metric
quality
Value
0.9037
Unit
accuracy (GSM8K fraction correct, 5-shot as configured by the reporter's shared eval settings)
Model
Qwen3.6-35B-A3B-NVFP4 (quant_algo W4A16_NVFP4 for mlp.experts)
Methodology
Qwen3.6-35B-A3B-NVFP4 served with --moe-backend flashinfer_b12x, same checkpoint and same eval settings as the Marlin reference row. The trace shows blackwell_sm12x.moe_micro_kernel.MoEMicroKernel (BF16 activations quantized to FP4 in-kernel, FP4 x FP4 MMA) rather than moe_w4a16_kernel.py (FP4 weights dequantized to BF16), because B12xMoEWrapper defaults activation_precision='fp4' and _normalize_quant_mode then resolves quant_mode='nvfp4'. 1192 correct. Latency 1080 s for the eval.
Measured by
third_party
Reproducible
yes
notes
The broken-path number; the Marlin reference on the same checkpoint scores 0.9136 (1205 correct, 833 s), so the cost of running a W4A16 checkpoint at W4A4 precision is 0.99 pp GSM8K (99 correct answers) here - small enough to pass any informal eyeball check, and large enough to matter on a MoE where the expert GEMMs dominate. Also note the path is SLOWER here (1080 s vs 833 s), so unlike the sm_121a Marlin W4A8-FP8 case this mis-dispatch does not even look like a win. Sibling reference: nvfp4-w4a16-marlin-moe-gsm8k. See gotcha flashinfer-b12x-moe-runs-w4a16-as-w4a4. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-rtx-pro-6000-blackwell). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

The broken-path number; the Marlin reference on the same checkpoint scores 0.9136 (1205 correct, 833 s), so the cost of running a W4A16 checkpoint at W4A4 precision is 0.99 pp GSM8K (99 correct answers) here - small enough to pass any informal eyeball check, and large enough to matter on a MoE where the expert GEMMs dominate. Also note the path is SLOWER here (1080 s vs 833 s), so unlike the sm_121a Marlin W4A8-FP8 case this mis-dispatch does not even look like a win. Sibling reference: nvfp4-w4a16-marlin-moe-gsm8k. See gotcha flashinfer-b12x-moe-runs-w4a16-as-w4a4. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-rtx-pro-6000-blackwell). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (1)