AirLLM
cuda, cpu, metal
Fields
id
engines/niche-airllmRepo
lyogavin/airllm
Languages
Python
License
Apache-2.0
First release
2023
Design goal
minimal-vramresearchrun-anything-anywhere
Scheduling
none
Backends
cudacpumetal
Formats
safetensors
Notable features
Layer-wise inference: the model is decomposed into per-layer files on disk and streamed onto the accelerator one layer at a time, so peak VRAM is roughly one layer plus activations rather than the whole model. This is the entire idea and it is genuinely architecturally interestingExplicitly no quantization, distillation or pruning required for the headline claim - the memory saving comes from not holding the model residentv3.0 (2026/06) added FP8 and streamed MoE support: DeepSeek-V3 671B on ~12GB, Qwen3-235B on ~3GB; 2026/07 claims Kimi K3 (2.8T) in 3.72GB of VRAM on one RTX 6000 Ada; 2026/08 claims Qwen3.8-Flash-Next (125B) in 5.95GB on an RTX 4090Per-expert streaming for MoE so only the experts a token routes to are loadedBlock-wise 4bit/8bit 'compression' of weights only, with the README correctly noting this is not quantization because activations are untouchedPrefetching to overlap disk load with compute, on by default (AirLLMLlama2 only per the README)v2.10.1 added CPU inference and non-sharded models; Apple Silicon/MLX path documented with a notebook2026/09 added streaming training: frozen weights streamed a layer at a time with adapters kept on GPU
Hardware caveats
This is a memory-capacity tool, not a throughput tool. The README gives you VRAM footprints and not one decode tok/s figure; the open issues are where speed is discussed and they are dire: '28.6 s/token on a 3B model' on Windows+NVIDIA, 'taking about 40 minutes to generate one sentence', 'VERY SLOW' on an AMD 9070 XT with ROCm 7.12, and an explicit '3x speedup compression claim is false/outdated'Requires roughly 2x the model size in HuggingFace cache disk. The README warns about this in the quickstart: 'the original model will first be decomposed and saved layer-wise. Please ensure there is sufficient disk space'Every generated token re-streams the whole model from disk. Disk I/O is the bottleneck by construction, so performance tracks your storage bandwidth, not your GPUROCm support is effectively unproven - the 9070 XT report is the only direct evidence and it is negativeWindows is a rough path (the 28.6 s/token report)No batching, no continuous batching, no paged KV: scheduling is 'none'. Each request pays the full layer-streaming costOnly safetensors/HF-format models; no GGUF, no AWQ/GPTQ/EXL2Marked draft because the README's headline VRAM numbers are self-measured by the maintainer with no methodology, and the speed claims are actively contradicted by user reports. The 35332 stars and last push 2026-10-03 show it is very widely used as a 'can I run this at all' demo
Best for
Answering 'can this model run on this GPU' when the answer is technically yes but practically miserable - a 70B on a 4GB card is a real capability that no other engine in this repo offersDemonstrating the layer-streaming idea, which is the correct mental model for why MoE experts can be streamed and why KTransformers-style CPU offload worksStreaming fine-tuning of a model that does not fit, keeping only adapters resident
Avoid for
Anything with a latency or throughput SLO - you will be serving minutes per sentenceConcurrent or production serving; there is no schedulerAny claim you have to defend with numbers - use GGUF quantization via [[llama-cpp]] instead and pay 4 bits instead of disk bandwidth
notes
The 35k stars are almost entirely from people confirming it starts. The value of the record is architectural, not operational: it is the clearest public demonstration that inference memory footprint and inference speed are separate problems. Confidence deliberately low; the VRAM claims are vendor self-measurements and the speed claims have public contradiction. Predecessor name: the repo lyogavin/Anima now redirects to this one. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0.0 months before this record), latest release v4.0.0 published 2026-09-05, 10 contributors (distinct committers including anonymous), open_issues_count 155 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 35377 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Notes
The 35k stars are almost entirely from people confirming it starts. The value of the record is architectural, not operational: it is the clearest public demonstration that inference memory footprint and inference speed are separate problems. Confidence deliberately low; the VRAM claims are vendor self-measurements and the speed claims have public contradiction. Predecessor name: the repo lyogavin/Anima now redirects to this one. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0.0 months before this record), latest release v4.0.0 published 2026-09-05, 10 contributors (distinct committers including anonymous), open_issues_count 155 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 35377 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.
Cross-references
This record references no other records — it is a leaf. It is cited by 1.