Fast Distributed Inference Serving for Large Language Models (arXiv 2305.05920)

Fields

id

sources/pap-sys-fastserve-arxiv

URL

https://arxiv.org/abs/2305.05920

Publisher

arXiv (Peking University)

Kind

paper

Published

2023-05-10

Accessed

2026-10-03

notes

arXiv cats cs.LG/cs.DC, no venue in comment field. Mechanism: token-granularity preemption plus a skip-join multi-level feedback queue scheduler. Distinct from FastServe for MoE serving, which is a different paper.

Notes

arXiv cats cs.LG/cs.DC, no venue in comment field. Mechanism: token-granularity preemption plus a skip-join multi-level feedback queue scheduler. Distinct from FastServe for MoE serving, which is a different paper.

Cross-references

This record references no other records.