inference/research
verifiedhigh · 0.92updated 2026-10-04

HeteGen: Efficient Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices

serving-systems · mlsys

Fields

id
papers/hetegen-heterogeneous-parallel-inference
arXiv
2403.01164
Venue
mlsys
Venue track
main
Year
2024
Category
serving-systems
Adoption
research-only
Authors
Xuanlei ZhaoBin JiaHaotian ZhouZiming LiuShenggan ChengYang You
Affiliations
National University of Singapore
Problem
On a device whose DRAM cannot hold the model, offloading weights from storage is I/O-bound, so latency collapses; offloading to a CPU adds PCIe transfer that dominates the compute of the layers themselves.
Mechanism
Heterogeneous CPU/GPU parallel inference: the model is offloaded across CPU and GPU, and each layer's forward pass is overlapped with the transfer of the next layer's weights, so I/O latency is hidden behind compute instead of serialized with it. The framing is that the two devices are complementary (CPU for I/O, GPU for compute) rather than one being a fallback for the other.
Speedup reported
Paper's own claim: outperforms state-of-the-art offloading solutions by up to 317% (i.e. ~4.2x) on an OPT-30B workload, on an NVIDIA A10 GPU paired with an Intel Xeon @ 2.30GHz, batch size 1 and sequence length 512.
Code
https://github.com/XuanleiZhao/HeteGen
Open weights
yes
notes
Added because disaggregated serving in this repo (papers/distserve, papers/splitwise, papers/rapid-power-aware-disaggregated-inference) only covers GPU-to-GPU prefill/decode split; HeteGen is the resource-constrained end of the same idea and is the record that says the split is not always between two GPUs. TITLE DISCREPANCY, kept honestly: the MLSys 2024 proceedings title is 'HeteGen: Efficient Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices' (read off the static proceedings.mlsys.org index) while the arXiv title OMITS the word 'Efficient'. Same paper; the proceedings title is used here as the citable one. Affiliation 'National University of Singapore' comes from the PDF first-page footnote ('1National University of Singapore') and is corroborated by OpenAlex; note the six-author list is a single-institution paper, which is unusual for this sub-area and corrects the common assumption that heterogeneous inference needs an industry lab. arXiv 2403.01164 comment field independently reads 'MLSys 2024'.

Notes

Added because disaggregated serving in this repo (papers/distserve, papers/splitwise, papers/rapid-power-aware-disaggregated-inference) only covers GPU-to-GPU prefill/decode split; HeteGen is the resource-constrained end of the same idea and is the record that says the split is not always between two GPUs. TITLE DISCREPANCY, kept honestly: the MLSys 2024 proceedings title is 'HeteGen: Efficient Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices' (read off the static proceedings.mlsys.org index) while the arXiv title OMITS the word 'Efficient'. Same paper; the proceedings title is used here as the citable one. Affiliation 'National University of Singapore' comes from the PDF first-page footnote ('1National University of Singapore') and is corroborated by OpenAlex; note the six-author list is a single-institution paper, which is unusual for this sub-area and corrects the common assumption that heterogeneous inference needs an industry lab. arXiv 2403.01164 comment field independently reads 'MLSys 2024'.

Cross-references

This record references no other records — it is a leaf. It is cited by 0.