AirLLM GitHub repository · inference-research

Fields

id

sources/niche-airllm-repo

URL

https://github.com/lyogavin/airllm

Publisher

Gavin Li

Kind

repo

Accessed

2026-10-03

notes

AirLLM README. Headline claim: 70B inference on a single 4GB GPU with no quantization, distillation or pruning, by decomposing the model layer-wise to disk and streaming layers. v3.0 (2026/06) added FP8 and claims DeepSeek-V3 671B on ~12GB and Qwen3-235B on ~3GB; 2026/07 claims Kimi K3 (2.8T) in 3.72GB on one RTX 6000 Ada. Requires 2x disk in the HF cache and warns about it explicitly. Block-wise 4bit/8bit 'compression' is described as a 3x speed-up; the README itself distinguishes it from quantization because only weights are compressed. Apple Silicon/MLX path documented. API metadata 2026-10-03: created 2023-06-12, last push 2026-10-03, v4.0.0 (2026-09-05), Apache-2.0, 256 open issues, 35332 stars.

Notes

AirLLM README. Headline claim: 70B inference on a single 4GB GPU with no quantization, distillation or pruning, by decomposing the model layer-wise to disk and streaming layers. v3.0 (2026/06) added FP8 and claims DeepSeek-V3 671B on ~12GB and Qwen3-235B on ~3GB; 2026/07 claims Kimi K3 (2.8T) in 3.72GB on one RTX 6000 Ada. Requires 2x disk in the HF cache and warns about it explicitly. Block-wise 4bit/8bit 'compression' is described as a 3x speed-up; the README itself distinguishes it from quantization because only weights are compressed. Apple Silicon/MLX path documented. API metadata 2026-10-03: created 2023-06-12, last push 2026-10-03, v4.0.0 (2026-09-05), Apache-2.0, 256 open issues, 35332 stars.

Cross-references

This record references no other records.