inference/research
verifiedhigh · 0.95updated 2026-10-04

Orca: A Distributed Serving System for Transformer-Based Generative Models

serving-systems · osdi

Fields

id
papers/orca-osdi22-iteration-scheduling
Venue
osdi
Venue track
main
Year
2022
Category
serving-systems
Adoption
in-production
Authors
Gyeong-In YuJoo Seong JeongGeon-Woo KimSoojeong KimByung-Gon Chun
Affiliations
Seoul National UniversityFriendliAI
Problem
Transformer autoregressive generation yields one token per request per forward pass, so a serving engine that schedules at request granularity cannot fill the GPU with work from a sequence that finishes early, and head-of-line blocking wastes most of the batch.
Mechanism
Iteration-level scheduling (later universally called continuous batching): the scheduler admits new requests into the RUNNING batch as soon as any individual sequence finishes, rather than waiting for the whole batch to drain. Paired with selective batching, which forms sub-batches only from requests whose operators actually need to run this iteration. Both are now default in every major engine.
Speedup reported
Paper's own claim: up to 36.9x higher throughput than NVIDIA FasterTransformer at the same latency level, and up to 11.2x better latency than FasterTransformer and vLLM at the same throughput, measured on GPT-3 175B (settings per the paper; baseline is FasterTransformer).
Code
https://github.com/microsoft/DeepSpeed/tree/master/deepspeed-inference
notes
THE paper the repo was missing on continuous batching, and a deliberate honest negative on identifiers: THIS PAPER HAS NO arXiv VERSION. Verified 2026-10-04 against export.arxiv.org/api/query with four different title/field searches, all returning zero entries; the only arXiv paper called 'Orca' is 2306.02707, Microsoft's distillation paper, which is a DIFFERENT work that happens to share the name. That is why arxiv_id is null here and why the pre-existing record papers/orca.json (2306.02707, arxiv-preprint 2023) is a legitimate separate entity rather than a duplicate. NAME COLLISION HAZARD: anyone searching this repo for 'orca' will hit the distillation paper first; this record is the one to cite for iteration-level scheduling / continuous batching. Affiliations from the USENIX page citation_author_institution meta tags (Seoul National University; FriendliAI), not from the paper PDF, because the paper was read via its proceedings page. Earlier records in this repo (flop-orca-osdi22, sched-orca-osdi22) already cited this work as a source with no paper record existing.

Notes

THE paper the repo was missing on continuous batching, and a deliberate honest negative on identifiers: THIS PAPER HAS NO arXiv VERSION. Verified 2026-10-04 against export.arxiv.org/api/query with four different title/field searches, all returning zero entries; the only arXiv paper called 'Orca' is 2306.02707, Microsoft's distillation paper, which is a DIFFERENT work that happens to share the name. That is why arxiv_id is null here and why the pre-existing record papers/orca.json (2306.02707, arxiv-preprint 2023) is a legitimate separate entity rather than a duplicate. NAME COLLISION HAZARD: anyone searching this repo for 'orca' will hit the distillation paper first; this record is the one to cite for iteration-level scheduling / continuous batching. Affiliations from the USENIX page citation_author_institution meta tags (Seoul National University; FriendliAI), not from the paper PDF, because the paper was read via its proceedings page. Earlier records in this repo (flop-orca-osdi22, sched-orca-osdi22) already cited this work as a source with no paper record existing.

Cross-references

Hardware relevance (1)