WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware

With low VRAM and a lot of host RAM, a static layer offload still copies every expert in the layer on every forward even though routing uses a few of them, and a predictor that prefetches the next experts does not hide that copy: at batch-1 decode there is no compute to overlap it with, so the speculative transfer just fights the demand transfer for the same PCIe link.

Treat resident experts and the KV cache as two working sets sharing one VRAM budget, and page only the experts a workload reuses. Expert masters stay pinned in host RAM. Each MoE layer gets a GPU scratch of C slots; the engine's existing expert_map is rewritten so a logical expert id points at a scratch slot holding the same bytes, and the unmodified fused-MoE kernel runs against that map. LRU decides who stays, because the regime is reuse-limited rather than overlap-limited. A miss copies one expert from the pinned host master into a victim slot; it does not copy the layer. For a prefill whose expert union exceeds C, tokens are grouped so each group fits the scratch and outputs are reassembled in token order; single-stream decode never takes that path. MV-WSA then splits the pageable budget (scratch plus KV pool, excluding the fixed non-expert slab) by equalizing latency removed per byte. An expert miss costs Me times the measured byte time; a KV miss costs a recompute. The split is clamped by an admission floor so the KV pool can still hold the requested concurrency times context. Offline, a short trace emits the expert cap and the KV pool size. Online, at a drained barrier, the controller measures peak KV occupancy, keeps KV just above that peak plus headroom, and gives the rest to the expert scratch, freeing the shrinking side before growing the other so the transient footprint never exceeds the budget. The plug-in patches three methods on vLLM's fused-MoE module through the general_plugins entry point: weight creation (pinned host masters, so load does not spike GPU memory), post-load (the C-slot scratch and the forward and inverse maps), and the forward. On the FP8 block-quantized path it pages four tensors per expert, the weights and their block scales, and forces the Triton backend so it bypasses DeepGEMM's load-time post-processing, which assumes the whole expert bank is resident.

Authors' own measurements, vLLM 0.11.2, eager mode, temperature 0. Prefetch negative, Section 4 and Figure 2, Qwen3-30B-A3B, single 24 GiB RTX 3090, PCIe 4.0, measured host-to-device about 25 GB/s: turning on speculative prefetch with an online co-activation predictor (about 0.5 next-expert precision on per-user traces) lowers decode from 5.47 to 3.37 tok/s at cap 8 and from 6.94 to 2.79 tok/s at cap 16, a 38-60% drop, and raises TTFT by 1.6-1.9x. A topic-coherent session reaches no higher steady decode than a diverse one (6.47 vs 6.82 tok/s). Only 44-48% of each layer's top-8 experts carry over between adjacent decode steps, so even a perfect last-step cache still streams about 1.8 GiB of expert weights per token, which at 25 GB/s caps decode at roughly 13-16 tok/s before any kernel overhead. Iso-VRAM decode, Table 2 and Figure 3, same 3090, BF16: Qwen3-30B-A3B (128 experts, top-8, about 57 GiB) at about 20 GiB budget is 7.62 tok/s at cap 16 against 4.17 tok/s for vLLM --cpu-offload-gb 50 (1.83x); at about 22 GiB, cap 32 is 9.93 against offload 44 GiB at 4.93 (2.01x). Caps 8/16/32 decode at 6.0/7.6/9.9 tok/s. Kimi-VL-A3B (64 experts, top-8, about 31 GiB): at about 14.7 GiB, cap 16 is 8.97 against offload 20 GiB at 5.00 (1.79x); at about 8.8 GiB, cap 8 is 6.69 against offload 24 GiB at 4.23 (1.58x). Table 3, real vllm serve, OLMoE-1B-7B, 8 GiB budget, AgentInstruct os, concurrency 4, iso-VRAM: MV-WSA (cap 21, 2.6 GiB KV) has median TTFT 2.9 s, decode 40.6 tok/s, turn wall 6.3 s; a 50-50 split (cap 18, 3.2 GiB KV) is 2.9 s, 29.2 tok/s, 7.0 s; a KV-blind expert-proportional split (cap 35, 0.5 GiB KV) is 19.8 s, 56.2 tok/s, 22.0 s; naive offload is 21.3 s, 30.7 tok/s, 25.5 s. Table 4, live resize, same 3090, Qwen3-30B-A3B, concurrency 4, 36 timed turns: KV peaks at about 44% of its allocation, the controller moves cap 32 to 35 and KV 2.79 GiB to 1.53 GiB with zero preemptions. On the db trace TTFT falls from 95.46 s to 75.05 s (1.27x), decode rises from 3.02 to 3.36 tok/s (1.11x), end-to-end from 1355.4 s to 1138.4 s (1.19x). On os, TTFT 89.31 s to 80.74 s, decode 3.17 to 3.14 tok/s, end-to-end 1237.1 s to 1155.0 s (1.07x). Capability checks, not speedups: Jamba-v0.1 (52B, 103 GiB BF16, 16 experts, top-2) on one 94 GiB H100 OOMs at full residency and the cpu-offload path asserts, while WiSP decodes at 5.73 tok/s at cap 4 and 9.80 tok/s at cap 8. MiniMax-M2 (229B FP8, 256 experts, top-8) on one 94 GiB H100 with a 256 GiB host cgroup: offload of 180 GiB or more OOMs the host, WiSP decodes at 3.70 tok/s with a 14 GiB expert scratch at cap 32; stock vLLM is faster once the host can hold the staged copy. The text says the scratch on that run is 7-14 GiB.

VENUE: the abs comment and the v2 HTML both say preprint, under review. The HTML carries an ICML style-file warning, which is a template choice, not an acceptance, and no Crossref proceedings deposit was looked up because the paper does not claim one. Year is the 2026 submission year. WHAT TRANSFERS to a hand-written stack that keeps attention and shared experts resident and streams routed experts from RAM. (1) Do not spend the overlap budget on speculative expert prefetch in single-stream decode. The paper's own predictor is accurate enough (about 0.5) and still slows decode by 38-60%, because the mandatory miss traffic already fills the PCIe link: on Qwen3-30B-A3B only 44-48% of a layer's top-8 experts repeat on the next token, so about 1.8 GiB still moves per token and 25 GB/s of host-to-device bandwidth caps the step near 13-16 tok/s. Overlap pays when a batch activates enough experts that the GEMM outlasts the copy, which is the regime this paper explicitly is not. (2) Page experts, not layers. A C-slot scratch plus an indirection map means the fused MoE kernel, the quant format, and the attention kernel stay untouched; only the map and the slot contents change. That is the piece a hand-written kernel can copy without taking vLLM's plugin. (3) The bytes given to the resident expert set are bytes taken from the KV cache. A KV-blind split that maximizes the expert cap (cap 35, 0.5 GiB KV) posts the fastest decode in Table 3 and the worst turn time, because prefill then spills. Size the expert scratch from the reuse-distance miss curve and leave a KV floor of concurrency times context. (4) FP8 experts are four tensors, not one: weights and block scales both have to move, and any kernel that rewrites the weight layout at load (DeepGEMM here) cannot run on a bank that is only partly resident. WHAT DOES NOT TRANSFER: the 2.01x is against static layer offload on one 3090 in eager mode, not against a stack that already fetches only the routed experts. The paper says static residency wins once the model fits, and it could not reach that crossover on 24 GiB. The plug-in has no CUDA graphs, and the per-layer paging decision sits on the Python hot path, so the overhead grows as the cap grows and there is less to page. The live 1.19x is one model family, two traces, the closed-form step rule rather than the full estimator, inside the in-process engine rather than the multi-process serve loop. ktransformers on its AMX path is not measured. No low-bit expert path of its own: the BF16 and FP8 bytes per expert are whatever the checkpoint already is.