inference/research
verifiedhigh · 0.88updated 2026-10-04

PowerInfer: Fast Large Language Model Inference with a Consumer-grade GPU

serving-systems · sosp

Fields

id
papers/powerinfer-consumer-gpu-serving
arXiv
2312.12456
Venue
sosp
Venue track
main
Year
2024
Category
serving-systems
Adoption
research-only
Authors
Yixin SongZeyu MiHaotong XieHaibo Chen
Affiliations
Shanghai Jiao Tong University
Problem
A consumer PC has a GPU with far less memory and bandwidth than a datacenter GPU, so an LLM that fits nowhere must either spill to CPU (PCIe-bound, very slow) or be shrunk to a lower-quality model; neither is acceptable if you want datacenter-class quality on commodity hardware.
Mechanism
Exploits the power-law distribution of neuron activation: a small set of 'hot' neurons is consistently activated across inputs while the rest are input-dependent 'cold' neurons. Hot neurons are kept resident on the GPU with FP16 weights; cold neurons are handled on the CPU with lighter precision. A neuron-outlier predictor decides hot/cold placement ahead of execution, and request batching is made proactive because placement quality depends on request composition.
Speedup reported
Paper's own claim: up to 7.1x higher throughput than a standard framework (FlexGen) on the same consumer GPU at comparable latency; a llama.cpp-derived baseline is also beaten. See also the PowerInfer-2 follow-up (arXiv 2406.06282) for the smartphone result - a different paper, not this record.
Code
https://github.com/PowerInfer/PowerInfer
Open weights
yes
notes
Added for the commodity-hardware end of the serving story: the repo's serving records (pagedattention-vllm, distserve, splitwise, orca-osdi22-iteration-scheduling) all assume a datacentre GPU fleet. Venue SOSP 2024 - the arXiv comment field reads 'SOSP 2024' and Semantic Scholar returns 'Symposium on Operating Systems Principles'. NOTE a year discrepancy worth recording: Semantic Scholar's year field is 2023 (the arXiv posting year, 2023-12-16), while the venue year is 2024. This record uses the VENUE year 2024, consistent with how whisper and other PMLR-published papers in this repo already handle preprint-vs-venue year. The SOSP proceedings page itself was not fetched (ACM DL returns 403 to automated clients), so the venue rests on the arXiv comment plus S2 agreement. Affiliation 'Institute of Parallel and Distributed Systems, SEIEE, Shanghai Jiao Tong University' read from the PDF title block.

Notes

Added for the commodity-hardware end of the serving story: the repo's serving records (pagedattention-vllm, distserve, splitwise, orca-osdi22-iteration-scheduling) all assume a datacentre GPU fleet. Venue SOSP 2024 - the arXiv comment field reads 'SOSP 2024' and Semantic Scholar returns 'Symposium on Operating Systems Principles'. NOTE a year discrepancy worth recording: Semantic Scholar's year field is 2023 (the arXiv posting year, 2023-12-16), while the venue year is 2024. This record uses the VENUE year 2024, consistent with how whisper and other PMLR-published papers in this repo already handle preprint-vs-venue year. The SOSP proceedings page itself was not fetched (ACM DL returns 403 to automated clients), so the venue rests on the arXiv comment plus S2 agreement. Affiliation 'Institute of Parallel and Distributed Systems, SEIEE, Shanghai Jiao Tong University' read from the PDF title block.

Cross-references

This record references no other records — it is a leaf. It is cited by 0.