inference/research
verifiedhigh · 0.85updated 2026-10-03

EleutherAI lm-evaluation-harness

none

Fields

id
engines/lm-eval-harness
Repo
EleutherAI/lm-evaluation-harness
Languages
Python
License
MIT
First release
2021
Design goal
minimal-deps
Scheduling
none
Backends
none
Formats
hfvllmggufllama.cppexllamav2exllamav3mlx
Notable features
Standard harness for zero-shot and few-shot benchmark scoring (MMLU, GSM8K, HellaSwag, IFEval, TruthfulQA, ARC, PIQA, WinoGrande, ...)Version-pinned task metrics matter: task versions and metric names (acc vs acc_norm, strict vs loose) change reported numbersUsed as the downstream-scoring path by several quantization papers while perplexity was computed separately by the inference engine
Hardware caveats
Not an inference engine - it scores, it does not serve. Any tok/s figure attributed to it is from a backend plugin, not from the harness itself
Best for
comparable downstream accuracy numbers across quantization formats
Avoid for
latency or throughput measurement
Docs
https://github.com/EleutherAI/lm-evaluation-harness
notes
Recorded because almost every quantization quality delta in this repo is an lm-eval number, and because the GGUF k-quant study pinned the exact harness version (v0.4.9.2) alongside its llama.cpp commit (b7600). That pairing matters: swapping harness versions has silently moved published GSM8K/MMLU/IFEval numbers before. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-09-14 (0.7 months before this record), latest release v0.4.13 published 2026-08-31, 460 contributors (distinct committers including anonymous), open_issues_count 1115 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 14129 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.

Notes

Recorded because almost every quantization quality delta in this repo is an lm-eval number, and because the GGUF k-quant study pinned the exact harness version (v0.4.9.2) alongside its llama.cpp commit (b7600). That pairing matters: swapping harness versions has silently moved published GSM8K/MMLU/IFEval numbers before. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-09-14 (0.7 months before this record), latest release v0.4.13 published 2026-08-31, 460 contributors (distinct committers including anonymous), open_issues_count 1115 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 14129 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.

Cross-references

This record references no other records — it is a leaf. It is cited by 2.