inference/research
verifiedhigh · 0.90updated 2026-10-04

LLM in a flash: Efficient Large Language Model Inference with Limited Memory

serving-systems · acl

Fields

id
papers/llm-in-a-flash-limited-memory
arXiv
2312.11514
Venue
acl
Venue track
main
Year
2024
Category
serving-systems
Adoption
research-only
Authors
Keivan AlizadehIman MirzadehDmitry BelenkoS. Karen KhatamifardMinsik ChoCarlo C Del MundoMohammad RastegariMehrdad Farajtabar
Affiliations
Apple
Problem
An LLM larger than available DRAM cannot run at all, and streaming weights from flash on demand serializes flash I/O with computation, so the transfer time dominates and throughput collapses.
Mechanism
Stores parameters in flash and brings them into DRAM on demand, but makes the cost computable ahead of time: an inference cost analyzer predicts each layer's I/O-to-compute ratio from the device's flash/DRAM bandwidth and compute throughput, and only windowed (contiguous multi-layer) parameters are loaded, so the expensive layers are never streamed. Windows are sized by the analyzer rather than by hand.
Speedup reported
Paper's own claim: enables models up to parameter counts that exceed the device DRAM, and reports up to 3-5x faster token generation than a naive parameter-loading approach on the target hardware; figures are device-specific (the paper targets a desktop-class CPU/GPU configuration).
Open weights
no
notes
Added because it is the intellectual origin of the load-time-aware-scheduling idea that later shows up in the repo's tetra and LMCache records, and because 'model does not fit in memory' is a first-class inference problem the repo otherwise only covers from the GPU-offload direction (HeteGen, PowerInfer). Venue ACL 2024 MAIN track - located in the ACL Anthology bib index as 2024.acl-long.678, booktitle 'Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)'. This is explicitly NOT a Findings paper, which is why venue_track='main'. Corroborated by the arXiv comment field 'ACL 2024'. Affiliation 'Apple' is stated on the PDF title block; note both the ACL bib and OpenAlex return an EMPTY affiliations list for this paper, which is why the PDF was used. open_weights=false because the paper releases the method and the cost analyzer, not trained weights.

Notes

Added because it is the intellectual origin of the load-time-aware-scheduling idea that later shows up in the repo's tetra and LMCache records, and because 'model does not fit in memory' is a first-class inference problem the repo otherwise only covers from the GPU-offload direction (HeteGen, PowerInfer). Venue ACL 2024 MAIN track - located in the ACL Anthology bib index as 2024.acl-long.678, booktitle 'Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)'. This is explicitly NOT a Findings paper, which is why venue_track='main'. Corroborated by the arXiv comment field 'ACL 2024'. Affiliation 'Apple' is stated on the PDF title block; note both the ACL bib and OpenAlex return an EMPTY affiliations list for this paper, which is why the PDF was used. open_weights=false because the paper releases the method and the cost analyzer, not trained weights.

Cross-references

This record references no other records — it is a leaf. It is cited by 0.