Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024)

Fields

id

sources/sched-sarathi-osdi24

URL

https://arxiv.org/abs/2403.02310

Publisher

arXiv

Kind

paper

Published

2024-03

Accessed

2026-10-04

Archived URL

https://arxiv.org/html/2403.02310v3

notes

Introduced chunked prefill and stall-free scheduling. Key results: 2.6x higher serving capacity for Mistral-7B on single A100, up to 3.7x for Yi-34B on two A100s, up to 5.6x for Falcon-180B with pipeline parallelism. The arXiv HTML was fetched and verified.

Notes

Introduced chunked prefill and stall-free scheduling. Key results: 2.6x higher serving capacity for Mistral-7B on single A100, up to 3.7x for Yi-34B on two A100s, up to 5.6x for Falcon-180B with pipeline parallelism. The arXiv HTML was fetched and verified.

Cross-references

This record references no other records.