Repo
turboderp-org/exllamav2
Design goal
low-latencythroughput-firstsingle-userquantization-coverageresearch
Scheduling
continuous-batchingprefix-cache
Formats
exl2gptqsafetensorshf
Notable features
EXL2 is its reason to exist: mixed per-matrix bitrates (2.0-8.0 bpw) with column-level importance remapping, so sensitive weight columns get more bits at the same average bitrate. The original README claims Llama2 70B on a single 24GB GPU at 2.55 bpw, and 13B models at 2.65 bpw within 8GB.Dynamic generator: dynamic batching, smart prompt caching, and K/V cache deduplication in one API.Paged attention via Flash Attention 2.5.7+ as of v0.1.0+.Q4 K-cache mode, with the README stating it performs better than the old FP8 cache; the dynamic generator does not support FP8 cache.TabbyAPI is the official recommended server (OpenAI-compatible, HF model download, embeddings, HF Jinja2 chat templates).
Hardware caveats
NVIDIA-only. Requires the CUDA Toolkit plus gcc (Linux) or Visual Studio Build Tools (Windows), and an appropriate PyTorch build. There is no CPU, ROCm, or Metal path.The repo is ARCHIVED. The README carries a note that the project is archived for now and development continues on ExLlamaV3. Use V3 for new work.Slow CPUs are called out as a potential bottleneck for these GPU-side kernels — ExLlamaV2 moves a lot of data per token and CPU-side work can gate it.Some early EXL2 models used no GQA, which effectively capped context at 2048 tokens. Not a current-model problem but a trap when picking old conversions off the Hub.All speed figures in the README are the author's own ('in my tests'), on a 3090Ti/4090, comparing against ExLlama V1 — not against llama.cpp or vLLM.
Best for
Getting a very large model into a single consumer NVIDIA GPU card — EXL2's variable-bpwinth trick is the whole point.Single-user local interactive use where decode tok/s on one card is what matters.2-3 bpw territory where generic quants fall apart and you want per-model calibration.Reference-quality community EXL2 conversions (turboderp, LoneStriker, bartowski are the named sources).
Avoid for
New projects — it is archived; start on ExLlamaV3 (EXL3) instead.AMD, Intel, Apple, or CPU-only hardware of any kind.Multi-user serving that needs paged KV across many concurrent sequences — paged attention exists but the project is single-user-oriented.Anyone needing a maintained dependency chain.
notes
Kept as a distinct record from exllamav3 because EXL2 vs EXL3 is a format break, not a version bump — a V2 checkpoint will not load in V3. Status 'verified' with the archived status recorded explicitly rather than marked 'deprecated', since the code still runs and the EXL2 model corpus is large and still in use; 'deprecated' would imply removal guidance the source does not make. Performance figures quoted in notable_features are the author's own claims and are not reproduced as benchmark records here because methodology (batch size, prompt/output length) is not specified in the README. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-03-04 (7.0 months before this record), latest release v0.3.2 published 2025-07-13, 57 contributors (distinct committers including anonymous), open_issues_count 161 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 4632 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. ||| DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - repositories, READMEs, docs and issue trackers were searched. See [[ev-production-evidence-is-scarce-and-nearly-always-single-site]] for the grading scheme; note that for an artifact-evaluation or paper-code release, non-deployment is documented intent rather than a negative finding.