The Embedder's Dilemma: LLMs Are Better, but at What Cost?

Teams are being pushed to swap embedding pipelines for LLMs without a controlled cost comparison.

Controlled cost-aware comparison of 10 LLMs across 6 families against 26 dedicated embedding models (118M-14B) over 37 tasks spanning classification, STS, clustering, pair classification and retrieval, measuring both quality and cost.

In aggregate the paradigms are effectively TIED - best LLM 77.6 vs best embedding model 77.2, a 0.4-point gap. But an LLM costs up to 1,431x more than a comparable-quality embedding model (USD 154 vs USD 0.11 per benchmark pass), open LLMs process tokens 2.5-736x more slowly on the same GPU, and reasoning tokens account for 28-81% of LLM inference cost. Baselines, hardware and the cost unit are all stated.

Venue COLM 2026 (arXiv comment). THE PAPER TO READ BEFORE UPGRADING AN EMBEDDING PIPELINE: it shows LLM embedders win on reasoning-heavy retrieval while losing on classification and cost-elsewhere, so the 'LLMs are better' framing is wrong. The two numbers that should change a serving decision are the 2.5-736x throughput gap and the finding that lower reasoning budgets preserve or improve retrieval quality - i.e. for most retrieval traffic the correct move is to lower the reasoning budget rather than to abandon the encoder. The Pareto frontier containing the leading embedding models is the actionable summary. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: colm has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.