Arithmetic intensity
MoE AI is worse than dense by exactly the activation sparsity factor, and that is the whole serving story. With E experts, top-k, total expert params N and batch B: each token touches k/E of the parameters, so flops = 2*B*N*(k/E) while bytes = 2*N (the entire expert weight set is what must be streamed to get k/E of the work done - a batch large enough to touch most experts does not reduce the bytes, because every expert is still needed by somebody). AI = B*(k/E) flop/byte. Worked against DeepSeek-V3's own config.json: 256 routed experts, 8 activated per token (k/E = 1/32), so AI = B/32. H100 SXM's ridge point is ~295 flop/byte, so an MoE decode step needs a batch of about 32*295 = 9,440 to reach compute-bound where the same-width dense model needs ~295. Confirmed in the shipped configs: DeepSeek-V3 256 experts/8 per token; DeepSeek-V2 160/6 (1/26.7); Qwen3-Next 512/10 (1/51.2, the most aggressive in this set); Kimi Linear 256/8; GPT-OSS-20b num_experts_per_tok 4; OLMoE-1B-7B 64/8 (1/8). The verification the task asked for: this is the mechanism by which a same-parameter-count MoE can be SLOWER to serve than a dense model. Per-token FLOPs are genuinely lower (k/E of the work), but the memory roof does not know the model is sparse - you pay the full weight footprint of all 256 experts at load time - so at any batch where the dense model would already be memory-bound, the MoE is more memory-bound, and the only way out is a batch ~32x larger. DeepSeek-V3 makes the point concretely by shipping first_k_dense_replace=3: the first 3 of 61 layers are dense FFN, so even DeepSeek's own MoE keeps a dense prefix. Shared experts make the arithmetic worse in the most-quoted case: DeepSeek-V3 has n_shared_experts=1 in ADDITION to 256 routed, so every token pays 8 routed + 1 always-on expert = 9/257 of the expert parameters, and DeepSeek-V2 pays 6 routed + 2 shared = 8/162.
Scales with
batch_size (AI = B*k/E - the dominant lever, and E/k times harder to move than in the dense case)num_experts and top_k (through k/E: DeepSeek-V3 1/32, DeepSeek-V2 1/26.7, Qwen3-Next 1/51.2, OLMoE 1/8)num_shared_experts (DeepSeek-V3's always-on expert adds 1/257 to every token's active expert set; DeepSeek-V2 adds 2/162)expert_intermediate_size (DeepSeek-V3: 2048 per expert per the technical report, vs 18432 dense intermediate in config.json - this is the fine-grained segmentation and it is why k must be large to reach the same per-token compute)expert_parallel_degree (sharding splits bytes across devices and adds an all-to-all)weight_dtype (the only lever that reduces the bytes at all, since the flop count is not the problem)num_dense_layers (DeepSeek-V3's first_k_dense_replace=3 layers are dense and contribute full-width bandwidth)
Hardware parameters
parameterpeak_flopseffectH100 SXM's ridge point is ~295 flop/byte, so an MoE decode step needs a batch of about 32*295 = 9,440 to reach compute-bound where the same-width dense model needs ~295.basisrecordedmagnitude295 flop/bytevalues_fornvidia-h100-sxm,amd-instinct-mi300x,nvidia-b200,google-tpu-v5e
Workarounds
Fused MoE kernels: gather tokens, run the expert GEMM and scatter back in a single kernel so the permuted activation never round-trips through HBMBlock-sparse GEMM formulations (MegaBlocks, arXiv 2211.15841) that handle expert-size dynamism without padding to the max expertExpert pruning and merging to shrink total MoE size (DeepSpeed-MoE reports up to 3.7x)int4/fp8 expert weights plus expert-parallel placement so each device's expert shard fits a larger active batch - with AI = B*k/E, halving weight bytes halves the batch needed to reach the ridgeGrouped GEMM / bmm layouts so each expert's GEMM has a usable M dimension rather than a token-thin matrixShared-expert isolation so common knowledge is not duplicated across 256 routed experts - this cuts total parameters for equal quality, which is the only intervention that improves the k/E ratio rather than working around it
notes
Design consequence, now with the arithmetic attached: a MoE model with the same parameter count as a dense model is not faster to serve - it is slower, because you must stream every expert weight to get k/E of the work done. DeepSeek-V3's 671B total / 37B active is the canonical case: the memory roof is set by 671B while the flops are set by 37B, and those two numbers are what a capacity planner and a FLOPs planner respectively need. Serving MoE well demands a high active batch, which means latency-optimized MoE serving and throughput-optimized MoE serving are different products. Hardware designers should read MoE inference as a bandwidth problem with a bigger weight footprint and a smaller flops-per-byte, not as a compute win. Interconnect choice matters twice over: once for weights staying local, once for the all-to-all. MegaBlocks' central point carries over to inference - padding every expert's token group to the largest expert wastes flops and bytes in exactly the regime that is already bandwidth-starved. DeepSeekMoE (arXiv 2401.06066) adds the reason fine-grained segmentation exists at all: keeping total and compute constant while splitting the FFN intermediate hidden dimension into more, smaller experts 'allows diverse knowledge to be decomposed more finely', and shared-expert isolation compresses common knowledge into always-on experts so 'redundancy among other routed experts will be mitigated'. The serving consequence of fine-graining is subtle and worth stating: it is what forces k up (DeepSeek-V3 activates 8 of 256 experts per token, not 2 of 8), and a larger k makes the router and the all-to-all more expensive while leaving k/E roughly where it was. So fine-grained experts buy quality at the cost of a busier interconnect, not at the cost of bandwidth.
HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.