MTEB: Massive Text Embedding Benchmark

Text embeddings were evaluated on a couple of STS datasets, so field progress could not be tracked and models did not generalise across task types.

A BENCHMARK, not a method: 8 embedding task types over 58 datasets and 112 languages, with a standard evaluation harness and a public leaderboard, used to evaluate 33 models. Its central empirical finding is that no method dominates across all tasks.

Benchmarks 33 models across 8 tasks, 58 datasets and 112 languages, and establishes that no particular text embedding method dominates across all tasks - i.e. the field has not converged on a universal method. No speedup claim; this is an evaluation artifact.

Venue EACL 2023 (crossref 'Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics'). Recorded because it changed how this field buys models: MTEB scores are the default acceptance gate for an embedding model, and its headline finding is the warning that a single benchmark score is not a reason to swap a production embedder. The hardware-relevant consequence is indirect but real - a benchmark that ranks encoders lets you trade model size against recall without running your own retrieval eval. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass): venue_track set to 'main' from the ACL Anthology bib index (anthology.bib, 131650 entries) by normalized title match. Evidence: anthology id 2023.eacl-main.148, booktitle 'Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics'. The Anthology id is the authoritative signal here and it is unambiguous, so this is an additive clarification of a track, not a change of venue - `venue` is untouched and no status change is implied. The reason this matters for inference readers: a Findings paper and a main-track paper at the same venue are not the same kind of evidence, and until this field existed the repo could not say which was which. VENUE CORRECTION + VENUE_TRACK 2026-10-04 (paper-fields pass, agent w3p): venue 'acl' -> 'other', venue_track 'main'. EVIDENCE: the ACL Anthology bib index (anthology.bib, 131650 entries, fetched 2026-10-04) contains this exact title as anthology id 2023.eacl-main.148 with booktitle 'Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics'. That is a proceedings record, so 'no venue found' was an under-claim, not a fact. BOTH VALUES RETAINED HERE PER THE REPO CONTRACT: the prior value was 'acl'. status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2210.07316). 2026-10-10: arXiv API 2210.07316 (axc.py: http://export.arxiv.org/api/query?id_list=2210.07316&max_results=1) comment is null. ACL Anthology 2023.eacl-main.148 confirms EACL 2023. Venue is EACL 2023 (schema enum gap: eacl missing from schemas/paper.schema.json). Venue remains other; status remains contested.