id
papers/moe-core-coordinated-expert-offload
Authors
Ke YangYongji GaoXushi LiKui LuoSicheng ZhangTianming ZhouKeyi LiuShufang LuAoxuan ChenJie MengJingchun GaoDan LiXinkai YouZhixiang XiaYan ShiYang LiuYanjia ZengLiangjun Feng
Problem
Routed experts live in host memory because the device cannot hold the pool, and the two phases want opposite fetches. Prefill activates a broad union of experts across many tokens, so a per-expert cache thrashes. Decode activates only top-k per step, so a fetch that starts after the gate sits on the critical path, and a uniform per-layer cache quota leaves the same layers under-covered on every workload.
Mechanism
Keep two physically disjoint device regions. Prefill uses two buffers, each sized to one full MoE layer, and swaps them: while layer l+1 computes, the released buffer loads every expert of layer l+2. The switch to full-layer staging happens when the prefill token count reaches floor(K_l / k_l), the resident slot count divided by the routing fanout; smaller prefills stay selective. Decode keeps a per-layer expert cache inside a fixed byte budget. Slot counts are not uniform: an offline replay of hit-rate curves moves slots in steps of delta-K from layers whose hit rate stays above the median and barely drops when shrunk, to layers at least tau_low below the median whose hit rate rises the most when grown. Cache contents start from a domain profile, a request hint naming the domain, filled with the K_l experts of highest mean router score on that domain's profiling set. During decode a routing-history score evicts: recent selection count over a window W, an EMA of selections, the current router score, and age since last selection, each normalized across the layer and combined with nonnegative weights. A miss is not waited on. Before layer l+1's real hidden state exists, the original router of layer l+1 is evaluated on layer l's hidden state, and the runtime prefetches only as many of the top predicted non-resident experts as the overlap window can hide: floor of (remaining compute of layer l plus attention of layer l+1, minus launch overhead, times H2D bandwidth, divided by expert bytes), further capped by free slots. An optional approximate path, off in the main numbers, substitutes a resident unselected expert for a low-score miss whose score is within alpha of the rank-(k+1) boundary, keeping the missed expert's routing coefficient. High-score misses still take the exact host-to-device copy.
Speedup reported
Authors' own numbers, batch size 1, Ascend 950PR with 128 GB physical NPU memory, a 150 GB host-memory limit, and an 84 GB NPU-memory cap. Models are DeepSeek-V4-Flash-W4A8 (uniform decode cache of 84 expert slots per layer; the nonuniform allocation uses 78 slots on most layers and 120 on layers 7, 9, 19, 31, and 37) and GLM-5.2-W4A8C8 (28 slots per layer). 200 requests per dataset, output cap 1K tokens for MoE-CORE and load-on-demand, 128 tokens for the vLLM prefetch baseline. AES and MTP are off in Table 1. Mean TPOT versus load-on-demand on DeepSeek: 38.0 vs 66.5 ms (GSM8K), 42.1 vs 62.9 (AIME 2026), 44.8 vs 62.1 (HumanEval), 41.5 vs 64.1 (MMLU), 40.3 vs 70.6 (LongBench v2), which the paper states as 1.39-1.75x. GLM mean TPOT falls from 229.1-244.8 ms to 206.6-220.5 ms. Prefetch depth on DeepSeek/GSM8K: 42.6 ms with none, 36.0 ms at one expert (15.5% lower), 35.9 ms at two, 36.5 ms at three. Removing the history-aware eviction raises that TPOT from 36.0 to 46.8 ms (30%). Domain profiles and the nonuniform slot allocation each move it by about 1%. AES raises effective expert availability from 92.43% to 99.972% but increases TPOT from 36.0 to 37.7 ms at batch 1. With AES on, batch 2, and MTP depth 2, TPOT is 21.5 ms.
notes
VENUE: the abs page has no conference comment and a Crossref title query on 2026-10-18 returned nothing, so venue is arxiv-preprint. The author list on the abs page names Dan Li twice; this record lists him once. Affiliations are not stated on the abs page. WHAT TRANSFERS to a hand-written stack that already streams routed experts from host RAM while attention and shared experts stay resident: (1) size the prefill staging buffer to one full expert layer, not to the union of a token's top-k, and overlap layer l+2's copy with layer l+1's compute, switching to that path only once the token count reaches floor(resident slots / top-k) — below that, selective residency moves fewer bytes; (2) do not split a fixed decode-cache byte budget evenly per layer, because the same layers are the hit-rate troughs on every workload they measured (layers 4 and 7 bottom-ten on all five, and seven layers strict local troughs on all five, at 84 slots on DeepSeek-V4-Flash-W4A8), and move slots only when the profiled hit-rate curve says the receiver gains more than the donor loses; (3) prefetch by running the real next-layer router on the current hidden state, and cap the copy at the experts the overlap window can actually hide — one expert captured 15.5% of TPOT (42.6 to 36.0 ms) and the second bought 0.1 ms while the third raised TPOT to 36.5 ms because three expert copies no longer fit under the remaining compute; (4) evict on router score plus a short and a long selection history, not LRU, which is the 30% TPOT gap (36.0 vs 46.8 ms) in their ablation, while the domain profile and the nonuniform quota are each about a 1% effect once that eviction is in place; (5) if a miss is still outstanding, a resident expert within a small score band of the top-k boundary can stand in and keep the missed expert's routing weight, which took effective availability from 92.43% to 99.972% but cost 1.7 ms of TPOT (36.0 to 37.7) at batch 1, so it is a tail-latency tool, not a throughput one. WHAT DOES NOT TRANSFER: the headline gap against vLLM prefetch (38-45 ms vs 1269 ms TPOT on DeepSeek, 207-221 vs 5942 ms on GLM) is not a kernel result. That baseline stages complete expert layers and was capped at 128 output tokens against 1K for the others, and the paper itself says a single layer's compute cannot hide a full-layer transfer during decode. The honest comparison is load-on-demand at the same 1K cap: 1.39-1.75x on DeepSeek and roughly 10% on GLM, where the larger experts leave only 28 resident slots. All of it is one Ascend 950PR under an 84 GB cap, not a PCIe GPU, and the paper does not jointly schedule the KV cache, so a long context that shrinks the expert region is unmeasured. No code is linked.