Defeating Nondeterminism in LLM Inference (Horace He et al., Thinking Machines Lab, 2025-09-10)

The clearest published statement of WHY batch invariance is the right oracle, and the source of the headline measurement: Qwen/Qwen3-235B-A22B-Instruct-2507, 1000 completions at temperature 0 on one prompt, 1000 tokens each, produced 80 UNIQUE completions - the most common occurring 78 times. All agreed for the first 102 tokens and first diverged at token 103. With their batch-invariant kernels enabled all 1000 completions were identical. Reframes the cause: the usual 'concurrency + floating-point non-associativity' story is wrong, because running the same matmul 1000x on the same data is bitwise identical; what varies is the BATCH-DEPENDENT REDUCTION STRATEGY (split-K, stream-K, split-KV/FlashDecoding, and small-batch split reductions). Their performance table, one GPU, Qwen-3-8B, 1000 sequences at 90-110 output tokens: vLLM default 26 s, unoptimized deterministic 55 s, + improved attention kernel 42 s. Kernels released as github.com/thinking-machines-lab/batch_invariant_ops. Also notes determinism is not a GPU-only property - CPU and TPU endpoints have the same batch-dependent-reduction cause.