MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism

MoE's sparsity makes inference MEMORY-intensive rather than compute-intensive at decode, which drops GPU utilization and raises operating cost - and it means the attention and FFN halves of a layer want different parallelism strategies that a single monolithic layer layout cannot give them.

DISAGGREGATES THE ATTENTION AND FFN MODULES WITHIN EACH LAYER, so the two are scaled independently with tailored parallelism strategies and can be deployed on HETEROGENEOUS hardware - the attention path is compute-bound and wants tensor parallelism, the FFN path is memory-bound and wants expert parallelism. The second mechanism is PING-PONG PIPELINE PARALLELISM: partition a request batch into MICRO-BATCHES and shuttle them between the attention side and the FFN side, which is what makes the disaggregation pay despite MoE sparsity - without a schedule that keeps both halves busy, splitting them just adds a hop. A high-performance M2N COMMUNICATION LIBRARY supports it, removing GPU-to-CPU data copies, group-initialization overhead and GPU synchronization that would otherwise dominate, and it exists specifically to minimize token-dispatch overhead across the disaggregation boundary.

Paper's own claim: up to 1.90x higher PER-GPU throughput than state-of-the-art solutions. The per-GPU qualifier is essential and must not be dropped: this is a utilization metric on disaggregated hardware, not an end-to-end request-latency number and not comparable to a single-node vLLM tok/s figure.

VENUE IS 'other' BECAUSE SIGCOMM IS NOT IN THE SCHEMA ENUM - the authoritative venue is SIGCOMM 2025, from the OpenReview note with venue 'SIGCOMM 2025' and venueid 'dblp.org/conf/SIGCOMM/2025'. The enum in schemas/paper.schema.json has no sigcomm entry, so the venue cannot be recorded honestly without either (a) claiming a venue the paper was not published at or (b) editing the schema, which is outside this pass's write scope. Recorded as 'other' with venue_track 'main' and the real venue in notes; the same convention the repo already uses for ColBERT and SPLADE. A future schema extension should add sigcomm. VERIFICATION CAVEAT, RECORDED BECAUSE IT IS NOT A CLEAN TITLE MATCH: the SIGCOMM note's title reads 'MegaScale-Infer: Efficient Mixture-of-Experts Model Serving with Disaggregated Expert Parallelism' - the PRE-PRINT title, not the current arXiv title. The identification therefore rests on the shared system name plus the identical mechanism (disaggregated expert parallelism) plus author-group overlap (Xin Jin appears on both the arXiv paper and the SIGCOMM record), not on a title string equality. This is the exact situation the repo's methodology rule anticipates, and it is recorded here rather than presented as a clean index hit. WHY IT IS A GAP: the repo's MoE serving work is DeepSpeed-MoE, MegaBlocks, Tutel/Flex, DeepSeek-V2/V3, Mixtral, OLMoE, MoEQuant, EAQuant and AlphaQ - all about MODELS or quantization. None disaggregates the layer internals for serving; the repo's disaggregation records are all prefill-vs-decode. Expert-vs-attention is a third axis and it is the one that matters when FFN weights dominate HBM traffic. VENUE RECLASSIFIED 2026-10-05 (schema authority pass, agent w5-schema): venue 'other' -> 'sigcomm', now that schemas/paper.schema.json carries a sigcomm member. The enum member was added only because this record existed and could be reclassified in the same commit - adding a venue nobody records is how this repo shipped 36 dead enum values. The venue was verified against the PUBLISHER'S OWN Crossref deposit, not an arXiv comment string, because this repo has already been burned by comment-derived venues. VENUE NOW VERIFIED FROM THE PUBLISHER'S DEPOSIT, superseding the OpenReview-only evidence in the note above: Crossref for DOI 10.1145/3718958.3750506 returns container-title 'Proceedings of the ACM SIGCOMM 2025 Conference', event "SIGCOMM '25: ACM SIGCOMM 2025 Conference", location 'São Francisco Convent Coimbra Portugal', pages 592-608, type proceedings-article, issued 2025-08-27, published-print 2025-09-08, parent volume 10.1145/3718958. That is a first-party ACM deposit and it settles the identification the note above flagged as resting on a title MISMATCH - the proceedings DOI carries the pre-print title, but the author list, the abstract ('disaggregates attention and FFN modules within each model layer', 'ping-pong pipeline parallelism', '1.90x higher per-GPU throughput') and the DOI parent volume all match this record exactly. AFFILIATION CORRECTED IN THE SAME PASS, because it came from the same deposit and was plainly wrong: this record previously claimed affiliation ['Rice University']. Crossref's author list for the SIGCOMM DOI gives 'School of Computer Science, Peking University, Beijing, China' (Zhu, Jin, Liu, Jin) and eight distinct ByteDance sites (Seattle, San Jose, Beijing, Shanghai, Hangzhou, Singapore). There is no Rice University author on the paper. This matters beyond bookkeeping: the paper's whole thesis is HETEROGENEOUS deployment across A-nodes and E-nodes, and attributing it to a university lab rather than ByteDance misdescribes where the production system ran. The OpenReview DBLP-mirrored note is still a valid second source and agrees on both venue and author list; the deposit simply upgrades it from mirror to first party. AFFILIATION EVIDENCE COMPLETED 2026-10-05 (wave 6, affiliation sweep, agent w6-papers). Wave 4 established that this paper's Rice University credit was an upstream metadata error; this pass confirms the positive side from the publisher's own deposit and records the FULL affiliation set rather than a two-entry summary. The ACM Crossref deposit for DOI 10.1145/3718958.3750506 carries a per-author affiliation array with EIGHT distinct strings (source w6p-crossref-sigcomm25-megascale-infer-affiliations): 'School of Computer Science, Peking University, Beijing, China', and seven ByteDance sites - Beijing, Hangzhou, San Jose, Seattle/China, Seattle/USA, Shanghai and Singapore. Seven of eight are ByteDance; the eighth is Peking University. No Rice University, and no other university. The field keeps 'School of Computer Science, Peking University, Beijing, China' and 'ByteDance Ltd' because the schema's array is an institution list and seven ByteDance city sites are one institution. The city detail is preserved here because it is not redundant: the deposit contains BOTH 'ByteDance Ltd, Seattle, China' and 'ByteDance Ltd, Seattle, USA', two DIFFERENT sites that render identically up to the country token. That is the general warning - an affiliation string is not an institution identifier, so it must not be deduplicated by city name, and a record that collapsed the ByteDance sites by city would have lost a real distinction. The venue note above remains accurate as written: this record's venue is now 'sigcomm' following the schema authority pass, and the identification is deliberately NOT a title-string match, since the proceedings title is the PRE-PRINT title ('Efficient Mixture-of-Experts Model Serving with Disaggregated Expert Parallelism') while the arXiv title is 'Serving Mixture-of-Experts at Scale with...'. Identification rests on the shared system name, the identical mechanism and author overlap (Xin Jin appears on both), which is stated in the record rather than presented as a clean index hit.