MoE gating: router GEMM, softmax/top-k, and the token scatter-gather around it

The router projection itself is tiny and bandwidth-bound: for t tokens, flops = 2*t*d_router*E while bytes = 2*d_router*E (weights, read once) + 2*t*d (activations), so AI is a small constant that grows only weakly with token count and is dominated by reading the router weights. DeepSeek-V3's d_router is hidden_size 7168 with E=256, so the router weight matrix is 1.83M elements - negligible against 671B. The softmax and top-k over E experts touch a t x E tensor (2*t*E bytes) for O(t*E) flops, again ~1 flop/byte. The expensive part of routing is not arithmetic at all: it is the permutation that moves each token to its k selected experts' devices, a network transfer whose cost is k*t*(hidden bytes)/link bandwidth. On DeepSeek-V3 that is k=8 x 7168 dims per token per layer. Three published mechanisms keep it bounded and each has a direct serving consequence. (1) DEVICE-LIMITED ROUTING (DeepSeek-V2 Sec 2.2.2): fine-grained expert segmentation means many experts are activated, so unrestricted top-K would scatter each token across many nodes; DeepSeek-V2 first picks the M devices holding the highest-affinity experts and does top-K only among those, 'we find that when M>=3, the device-limited routing can achieve a good performance roughly aligned with the unrestricted top-K routing'. DeepSeek-V3 states the shipped value: each token 'will be ensured to be sent to at most 4 nodes'. So the per-token all-to-all fan-out is capped at 4 hops regardless of k=8 - routing cost is bounded by M, not by k. (2) GROUPED ROUTING: DeepSeek-V2/V3 configs carry n_group 8 and topk_group 4 (DeepSeek-V3) / 3 (DeepSeek-V2), i.e. experts are partitioned into 8 groups and only 4 groups are eligible per token before top-k - this reduces the router's effective selection width and is the mechanism behind the device-limited behaviour. (3) BALANCE LOSSES: three auxiliary losses (expert-level L_ExpBal = alpha_1 * sum_i f_i*P_i, device-level L_DevBal over D expert groups, and communication-balance L_CommBal) that shape the router during training so that inference-time load is even. DeepSpeed-MoE (arXiv 2201.05596) reports up to 7.3x better latency/cost than prior MoE inference systems, and the gains come from exactly these system-level mechanisms rather than from the router math.

Design consequence: routing is a network problem wearing an arithmetic costume. Hardware and interconnect choices for MoE should be judged on all-to-all latency and whether expert parallelism can be laid out to keep most experts resident on one device, not on router FLOPS. Engine authors should expect router cost to show up as bubbles and jitter rather than as steady compute time, and should measure MoE decode with expert-parallel communication visible. The auxiliary-loss question has a concrete answer now: the load-balancing auxiliary loss DOES change real-world token counts, because balancing f_i against P_i forces the router to spread assignments, which changes how many distinct experts a batch touches - and distinct-expert count is exactly what determines how many expert weight shards must be streamed, so it feeds straight back into moe-experts' AI = B*k/E. DeepSeek-V3 abandoned the auxiliary loss for this reason and states the motivation directly: 'too large an auxiliary loss will impair the model performance', and their bias-term scheme 'achieves better performance than models that encourage load balance through pure auxiliary losses'. The serving reading of that: the aux loss is a training-time intervention whose cost lands at inference as expert-access locality. A router that is free to specialise (no aux loss) produces more skewed, more predictable expert loads - which DeepSeek-V3 reports shows 'greater expert specialization patterns than the auxiliary-loss-based one' - and predictable load is what lets you shard experts so that a token's k targets are usually on few devices. A caveat worth carrying: DeepSeek-V2 pairs its three balance losses with device-level token dropping at capacity factor 1.0, dropping lowest-affinity tokens per device, so the load balance it trains for is partly enforced by dropping - at inference it is optional ('we can flexibly decide whether to drop tokens during inference according to the efficiency requirements'), which means published MoE throughput numbers are not comparable unless you know whether token dropping was enabled. INFERENCE-TIME DISPATCH PATH, checked 2026-10-03 for coverage. Verdict: the arithmetic of dispatch and combine IS covered by deepep-expert-parallel-dispatch-combine (class kv_transfer, bound interconnect) and the expert GEMM by deepgemm-grouped-masked-moe-gemm, so there is no missing FLOP record here. What this record adds is the inference-only part that training-shaped descriptions omit. (a) Dispatch is a PER-TOKEN, PER-STEP operation at serving batch B, whereas at training time it runs on a large fixed token batch and is amortised against a big backward pass - the same bytes, but on a step whose GPU work is B tokens' worth of experts, so its RELATIVE cost is far higher in decode. Volume per step per layer is B x k x d x 2 bytes outbound and the same inbound on combine; at B=32, k=8, d=7168 (DeepSeek-V3's geometry) that is 32 x 8 x 7168 x 2 = 3.67 MB each way per layer, so ~220 MB per step across 61 layers against an expert weight stream of 2*37e9 active = 74 GB per step - dispatch is 0.3% of the bytes but sits on a DIFFERENT resource (the interconnect), and unlike HBM traffic it is latency-shaped, so it shows up as bubbles and jitter rather than as bandwidth saturation. (b) The capacity/token-dropping mechanism is an INFERENCE decision, not a training one: DeepSeek-V2 pairs its balance losses with device-level token dropping at capacity factor 1.0 and states dropping is optional at inference ('we can flexibly decide whether to drop tokens during inference according to the efficiency requirements'), so the serving operator chooses between capacity waste (padding every expert to the max, MegaBlocks' point) and quality loss (dropping tokens). (c) The decode path specifically avoids a host round trip: DeepEP's do_cpu_sync=False decode path allocates outputs to configured capacity and exposes psum_num_recv_tokens_per_expert so the downstream grouped GEMM can find valid ranges, instead of synchronising to the host to learn the dispatch result. So the inference dispatch distinction is: at decode, dispatch is interconnect-latency and host-synchronisation-sensitive, not flop-sensitive, and the single highest-leverage fix is removing the CPU sync. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.