MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs

MoE's low per-token FLOPs make it attractive on paper, but its size puts it out of reach on anything below a high-end GPU, so in practice the whole point of the architecture is unavailable to the people who would benefit most from it.

A CPU-GPU-I/O PIPELINING schedule called CGOPipe, paired with PAGED WEIGHTS, so that weights, CPU compute, GPU compute and I/O are all kept busy simultaneously instead of being serialized into phases. The second contribution is more useful to a reader than the system: a HIERARCHICAL ROOFLINE MODEL (HRM) that predicts achievable throughput from a memory hierarchy instead of requiring measurement, and a policy search driven by it. HRM is the transferable idea - it is the mechanism-level equivalent of the repo's existing FLOP-class records, applied to a three-level memory system, and it is why the paper can claim policies beating prior systems without a per-hardware tuning campaign. Paged weights are the enabler: they make expert weights pageable, which is what lets I/O overlap compute at all.

Paper's own claim: up to 10.3x higher throughput than state-of-the-art offloading-enabled LLM inference systems for Mixtral 8x7B on a SINGLE T4 GPU (16 GB); when theoretical throughput is GPU-memory-bound it reaches the throughput upper bound with 2-3x less CPU memory; and it supports Mixtral 8x22B and DBRX on 2-4 T4s. The 10.3x is against OFFLOADING-ENABLED baselines, which are a weak and fast-moving class - against a modern tensor-parallel deployment on one T4 the gap is smaller, and the paper's scope is not that comparison.

RECORDED AS arxiv-preprint WITH venue_track 'unknown' - VENUE SEARCH EXPLICITLY NEGATIVE, AND THIS IS THE MOST LIKELY RECORD IN THE BATCH TO HAVE BEEN GIVEN AN INVENTED VENUE. MoE-Lightning is widely cited and has an ML-systems venue history, so it is precisely the paper where 'oh that was an OSDI/MLSys paper' would be the easy wrong answer. WHAT WAS CHECKED: OpenReview holds ONE record for this title and it is the arXiv deposit itself - venue 'CoRR 2024', venueid 'dblp.org/journals/CORR/2024' - which is not a venue claim about publication. Corpus checks, all negative: PMLR v267 (ICML 2025) contains no MoE-Lightning entry; MLSys 2025 (61 papers) and MLSys 2026 (135 papers) contain none; NeurIPS 2024 (4493 papers) contains none. arXiv 2411.11217's comment field is EMPTY. So arxiv-preprint/unknown with confidence 0.75 is the honest record. WHY IT IS STILL A GAP WORTH FILLING: the repo has no MoE weight-offload paper with a performance MODEL in it, and HRM is the artifact a capacity planner could actually reuse. HARDWARE FIELD DELIBERATELY DOES NOT NAME THE PAPER'S OWN GPU: the testbed is a single NVIDIA T4 (16 GB), and this repo has NO T4 accelerator record, so rather than invent an id or substitute an H100 - which would silently misrepresent the paper's own platform - the field lists only accelerators that exist in data/accelerators/ (nvidia-a10, nvidia-a100-80gb-sxm4) and this note states the real one. Anyone adding a T4 record should repoint this field at it. VENUE CORRECTED 2026-10-05 (wave 6, agent w6-papers): arxiv-preprint/unknown -> ASPLOS 2025, status verified, confidence 0.75 -> 0.88. WAVE 4 SAID OF THIS RECORD: 'RECORDED AS arxiv-preprint ... DESPITE BEING WIDELY CITED - and the reason is worth stating because it is the case the task's rules exist for ... THIS IS THE MOST LIKELY RECORD IN THE BATCH TO HAVE BEEN GIVEN AN INVENTED VENUE.' The guess it declined to make was correct, and it would have been correct by luck only - the venue is ASPLOS, which is a venue almost nobody would guess for a MoE-inference paper and which is absent from all four corpora wave 4 swept. THE PUBLISHER'S OWN DEPOSIT (source w6p-crossref-asplos25-moe-lightning): ACM Crossref DOI 10.1145/3669940.3707267; container 'Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1'; event "ASPLOS '25: 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems", location Rotterdam Netherlands, acronym ASPLOS '25, sponsors SIGPLAN / SIGOPS / SIGARCH; pages 715-730; type proceedings-article; issued and published 2025-03-30. WHY WAVE 4'S SEARCH WAS EXECUTED CORRECTLY AND STILL MISSED IT - the generalisable lesson, and the reason this is worth more than a venue field. Wave 4 checked OpenReview, PMLR v267 (ICML 2025), MLSys 2025 and 2026, and NeurIPS 2024. Every one of those checks was right and every one of them has a blind spot: ASPLOS proceedings are deposited with ACM under a DOI and appear in NONE of the OpenReview index, PMLR, or the four conference indexes swept. A negative result across four corpora is only as strong as the union of what those corpora index. For a paper outside ML/NeurIPS-style venues, the ACM DL via Crossref is the channel, and nobody was looking there. This is the same shape as the PMLR-volume trap already logged in this repo, inverted: there the index gave a confidently wrong venue; here the index gave a confidently EMPTY answer. IDENTIFICATION IS EXACT, AND THE RAW TITLE WOULD HAVE FAILED A STRING CHECK: ACM renders 'MoE-Lightning' with the L of Lightning as an \textsc small-caps span, so the deposit's title string is literally 'MoE-L\n <scp>ightning</scp>\n : High-Throughput MoE Inference on' - multi-line, tagged, and not equal to the arXiv title under any normalised comparison. Identity rests on (a) the distinctive subtitle 'High-Throughput MoE Inference on Memory-constrained GPUs', (b) all NINE author names in arXiv order - Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E. Gonzalez, Matei Zaharia, Ion Stoica - matching the arXiv API response for 2411.11217, and (c) a plausible ASPLOS page range. Author overlap ALONE would have been insufficient here: the nine-name list also contains Ion Stoica and Joseph E. Gonzalez, who appear on many Berkeley systems papers. AFFILIATIONS CORRECTED IN THE SAME PASS, from the same deposit, which carries PER-AUTHOR affiliations: Stanford University (Ying Sheng), UC Berkeley (Cao, Shu Liu, Griggs, Schafhalter, Xiaoxuan Liu), and UC Berkeley again as the slightly different string 'UC Berkeley, Berkeley, USA'. This record listed UC Berkeley alone - partially correct, missing Stanford. Field now reads UC Berkeley + Stanford University. The three raw strings are 'Stanford University, Palo Alto, USA', 'UC Berkeley, Berkeley, CA, USA' and 'UC Berkeley, Berkeley, USA'; the last two differ only by the state token, so the field lists the INSTITUTION once.