DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference (arXiv 2401.08671)

Fields

id

sources/pap-sys-fastgen-arxiv

URL

https://arxiv.org/abs/2401.08671

Publisher

arXiv (Microsoft)

Kind

paper

Published

2024-01-09

Accessed

2026-10-03

notes

arXiv cats cs.PF/cs.LG, no venue in comment found. Headline claim from the abstract: up to 2.3x higher effective throughput, 2x lower average latency. Mechanism: Dynamic SplitFuse — composing prompt and generation work into a single dynamically-split inference pass. Venue NOT verified; recorded as arxiv-preprint.

Notes

arXiv cats cs.PF/cs.LG, no venue in comment found. Headline claim from the abstract: up to 2.3x higher effective throughput, 2x lower average latency. Mechanism: Dynamic SplitFuse — composing prompt and generation work into a single dynamically-split inference pass. Venue NOT verified; recorded as arxiv-preprint.

Cross-references

This record references no other records.