Qwen3 Technical Report (arXiv:2505.09388, Qwen / Alibaba) - the authors' own evaluation tables for Qwen3-8B and the Qwen3 family

FETCHED AND READ 2026-10-08: arXiv abs page plus the full HTML at https://arxiv.org/html/2505.09388v1 (135k chars, tables transcribed in full). arXiv:2505.09388, submitted 14 May 2025, An Yang, Baosong Yang ... Jingren Zhou (60 authors). The paper contains the Qwen3-8B reference numbers that the vLLM issue #56770 records quote as their BF16 baseline, from the paper's OWN tables - which is what makes the third-party figure independently checkable rather than self-reported. Table 17 (Thinking, Qwen3-8B vs Qwen3-4B): MATH-500 97.4, AIME'24 76.0, AIME'25 67.3, GPQA-Diamond 62.0, ZebraLogic 84.8, AutoLogi 89.1, BFCL v3 68.1, LiveCodeBench v5 57.5, CodeForces 1785/95.6%, Multi-IF 71.2, INCLUDE 67.8, MMMLU 74.4, MT-AIME2024 65.4, PolyMath 42.7, MLogiQA 69.0. Table 18 (Non-thinking, Qwen3-8B): MATH-500 87.4, AIME'24 29.1, AIME'25 20.9, GPQA-Diamond 39.3. Table 12 (Base models, Qwen3-8B Base): MMLU 76.89, MMLU-Redux 76.17, MMLU-Pro 56.73, SuperGPQA 31.64, BBH 78.40, GPQA 44.44, GSM8K 89.84, MATH 60.80, EvalPlus 67.65, MultiPL-E 70.73, MBPP 69.80, CRUX-O 62.00. The paper's shot conventions, from its Evaluation Setup: MMLU/MMLU-Redux/MMLU-Pro 5-shot (MMLU-Pro and MMLU-Pro 5-shot CoT), BBH 3-shot CoT, SuperGPQA 5-shot CoT, GPQA 5-shot CoT, GSM8K 4-shot CoT, MATH 4-shot CoT, EvalPlus 0-shot. IMPORTANT BOUNDARY, recorded because it is the difference between a useful citation and an over-read one: these are the PAPER'S harness results on the authors' own evaluation stack, and the vLLM-issue records hold numbers produced by a DIFFERENT harness (vLLM + lm-eval style settings, 8x B200, same prompts reported by the reporter). So the paper corroborates the order of magnitude and the ranking (BF16 Qwen3-8B scores ~97 on MATH-500 and ~62 GPQA-Diamond in thinking mode) but is NOT the same measurement - the paper does not run AIME-24/25, GSM8K and MATH-500 together and does not average them, so the 82.36 figure itself remains the reporter's own construction on their own harness. Cited for exactly that: as the authors' primary document for the model's reported quality.