Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5)

Embedding models supervised on a single task or a small labelled set failed to generalise to retrieval.

Trains a BERT-family encoder contrastively against a curated large-scale text PAIR dataset (CCPairs) with weak supervision - the positive signal is that two texts co-occur, not a human relevance label. Adds an average-pooling layer for the embedding.

E5 is the first model to beat the BM25 baseline zero-shot on the BEIR retrieval benchmark without labelled data; fine-tuned it beats MTEB-best models using 40x fewer parameters. Quality claims on MTEB/BEIR; no latency or hardware disclosure.

VENUE: Semantic Scholar records this title's venue as arXiv.org only, and crossref finds no proceedings entry - cited here as a preprint rather than guessed into ACL/EMNLP. The practical reason E5 and its descendants are everywhere: a single 110-330M encoder serving retrieval, classification and clustering beats much larger alternatives, so the cheapest way to add embeddings to a system is to run one of these rather than an LLM. That is exactly the trade the 'Embedder's Dilemma' paper quantifies. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.