LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

Decoder-only LLMs were not used for text embeddings despite being the best available pretrained weights.

Three steps to turn any decoder-only LLM into an encoder: enable BIDIRECTIONAL ATTENTION, train with MASKED NEXT-TOKEN PREDICTION, then apply unsupervised contrastive learning (SimCSE-style). Applied to 4 LLMs from 1.3B to 8B parameters.

Outperforms encoder-only models by a large margin on word-level tasks and reaches new unsupervised state of the art on MTEB; combined with supervised contrastive learning it is state of the art among models trained only on public data as of May 2024. The paper reports NO inference-cost comparison against a same-size encoder - a critical omission, since an 8B bidirectional encoder is roughly an order of magnitude more expensive per query than a 110M encoder.

Venue COLM 2024 (arXiv comment). The important reading is the COST ONE, not the quality: this work makes a decoder LLM into an encoder, which means the per-query cost jumps to decoder-LLM scale. For high-QPS retrieval that is usually the wrong trade, and the Embedder's Dilemma paper below quantifies exactly how wrong. LLM2Vec is well suited to low-QPS, high-value embeddings (semantic caching keys, RAG over small corpora, agent memory) rather than to a hot query path. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: colm has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.