BGE Reranker v2 M3

RERANKER — the hardware profile is the important part. A cross-encoder concatenates query and passage into one sequence and scores it with a single linear head. N passages means N INDEPENDENT forward passes over (query+passage) pairs. There is no KV reuse across passages, because each pair is a different token sequence: the query prefix cannot be cached the way it is in a decoder's shared prompt. So a 100-passage rerank at 512 tokens is 100 full forward passes of 512 tokens, and cost scales linearly with the number of passages — unlike a decoder where prompt cost is amortized across the batch. kv_cache_bytes_per_token = 0 and the KV formula does not apply; this is prefill-like and COMPUTE-bound, not memory-bound. That is the structural difference from a bi-encoder like bge-m3: bge-m3 encodes each document once, bge-reranker-v2-m3 re-encodes every (query, passage) pair. The usual serving pattern is therefore retrieve wide with a cheap bi-encoder, then rerank only the top 50-100 — the reranker's cost is what forces that funnel. params_b 567755777 is a safetensors total (not derived). Same backbone as bge-m3 (XLM-RoBERTa, hidden 1024, 24 layers, vocab 250002, 8194 context) but Apache-2.0 rather than MIT. 17.1M downloads. GGUF (gpustack, cstr, smarttasks) and ONNX (onnx-community, kftof avx2 int8) ports exist — the int8 AVX2 builds are the realistic way to run this on CPU at usable latency. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=cross-encoder. REQUIRED KERNEL CLASS: N independent full forward passes over (query + passage) with no cache reuse and no early exit - flop/cross-encoder-reranking derives AI = t at batch 1 (544 flop/byte for BERT-base geometry at t=544, i.e. 1.84x ABOVE H100's 295.4 ridge) and AI = N*t when batched. It is compute-bound at batch 1, which makes buying FLOPS rather than bandwidth the correct procurement decision, and it is the one common serving workload where the usual quantization advice inverts: quantization helps only because it doubles peak FLOPS, not because it cuts bytes. Cost scales as candidates x sequence length. SUPPORTED BY (cited): vLLM pooling's cross-encoder row list is explicit - BertForSequenceClassification, GemmaForSequenceClassification (BAAI/bge-reranker-v2-gemma), GteNewForSequenceClassification, LlamaBidirectionalForSequenceClassification, ModernBertForSequenceClassification, Qwen2ForSequenceClassification (mixedbread-ai/mxbai-rerank-base-v2), Qwen3ForSequenceClassification (Qwen/Qwen3-Reranker-0.6B), RobertaForSequenceClassification, XLMRobertaForSequenceClassification (BAAI/bge-reranker-v2-m3) - with per-model score templates shipped for the rerankers that need a specific prompt format [[vllm-pooling-models]]. TEI has reranking first-class since v0.4.0 for CamemBERT, RoBERTa, XLM-RoBERTa and GTE sequence-classification cross-encoders at /rerank [[hf-text-embeddings-inference]]. SGLang splits the class in two and says so: 'Cross-encoder rerank models: run with --is-embedding (embedding runner)' vs 'Decoder-only rerank models: run WITHOUT --is-embedding and use next-token logprob scoring (yes/no)' - and its BGE-reranker row carries a hard kernel constraint, 'Currently only support attention-backend triton and torch_native', with the example launch passing --attention-backend triton, --disable-radix-cache and --chunked-prefill-size -1 [[mdl2-sglang-embedding-rerank-models]]. THE CLASS BOUNDARY THAT DECIDES ENGINE CHOICE: a cross-encoder is NOT late interaction. ColBERT-style multi-vector scoring is a different kernel class - a MaxSim over token-level embedding matrices, bandwidth- and capacity-bound on the index rather than compute-bound in the encoder, scaling with corpus size rather than batch size [[colbert-reference]]. No mainstream generation engine here serves it except vLLM, whose token_embed pooling type plus late-interaction MaxSim scoring is the only first-class ColBERT endpoint in the repo [[vllm-pooling-models]]. TEI cannot: its pooling options are cls, mean, splade and last-token, with no token-wise output mode, so 'a ColBERT model cannot be served correctly here' [[hf-text-embeddings-inference]]. llama.cpp-embedding-server cannot either: --pooling none gives raw per-token output, 'which is a building block for ColBERT but not a served ColBERT' [[llama-cpp-embedding-server]]. BATCHING TRAP, cited: reranking through Triton's sequence batcher requires modelling the query-document pair as ONE stateful unit rather than two independent sequences [[triton-inference-server]] - get that wrong and the pair is scored separately.