Mixedbread mxbai-rerank-large-v1

CROSS-ENCODER RERANKER: kv_cache_bytes_per_token = 0 and the KV formula does not apply — one bidirectional pass per pair, no cache. Cost model is the same as bge-reranker-v2-m3 and it is the point worth stating plainly: N passages = N independent forward passes over N (query+passage) sequences with NO KV reuse between them, because the query text sits inside each pair's sequence and cannot be cached across pairs. So rerank cost is linear in (number of passages x pair length) and COMPUTE-bound. The prefix is reused by the tokenizers but not by the model. Distinctive: DeBERTa-v2 backbone, not BERT or XLM-R — vocab 128100 (vs BERT's 30522), which is where the extra ~100M params over bge-large (435062785 vs 335142400) come from. Same hidden 1024 / 24 layers / intermediate 4096. 512 max_position_embeddings means long passages must be truncated before reranking, which is a real quality cliff on long-document search and the main reason bge-reranker-v2-m3 (8194 context) displaced this model for that use case. Apache-2.0. 44k downloads. ONNX ports exist (keisuke-miyako int8/f16, corto-ai); NO GGUF port found, so llama.cpp-based CPU reranking needs a different model. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=cross-encoder. REQUIRED KERNEL CLASS: N independent full forward passes over (query + passage) with no cache reuse and no early exit - flop/cross-encoder-reranking derives AI = t at batch 1 (544 flop/byte for BERT-base geometry at t=544, i.e. 1.84x ABOVE H100's 295.4 ridge) and AI = N*t when batched. It is compute-bound at batch 1, which makes buying FLOPS rather than bandwidth the correct procurement decision, and it is the one common serving workload where the usual quantization advice inverts: quantization helps only because it doubles peak FLOPS, not because it cuts bytes. Cost scales as candidates x sequence length. SUPPORTED BY (cited): vLLM pooling's cross-encoder row list is explicit - BertForSequenceClassification, GemmaForSequenceClassification (BAAI/bge-reranker-v2-gemma), GteNewForSequenceClassification, LlamaBidirectionalForSequenceClassification, ModernBertForSequenceClassification, Qwen2ForSequenceClassification (mixedbread-ai/mxbai-rerank-base-v2), Qwen3ForSequenceClassification (Qwen/Qwen3-Reranker-0.6B), RobertaForSequenceClassification, XLMRobertaForSequenceClassification (BAAI/bge-reranker-v2-m3) - with per-model score templates shipped for the rerankers that need a specific prompt format [[vllm-pooling-models]]. TEI has reranking first-class since v0.4.0 for CamemBERT, RoBERTa, XLM-RoBERTa and GTE sequence-classification cross-encoders at /rerank [[hf-text-embeddings-inference]]. SGLang splits the class in two and says so: 'Cross-encoder rerank models: run with --is-embedding (embedding runner)' vs 'Decoder-only rerank models: run WITHOUT --is-embedding and use next-token logprob scoring (yes/no)' - and its BGE-reranker row carries a hard kernel constraint, 'Currently only support attention-backend triton and torch_native', with the example launch passing --attention-backend triton, --disable-radix-cache and --chunked-prefill-size -1 [[mdl2-sglang-embedding-rerank-models]]. THE CLASS BOUNDARY THAT DECIDES ENGINE CHOICE: a cross-encoder is NOT late interaction. ColBERT-style multi-vector scoring is a different kernel class - a MaxSim over token-level embedding matrices, bandwidth- and capacity-bound on the index rather than compute-bound in the encoder, scaling with corpus size rather than batch size [[colbert-reference]]. No mainstream generation engine here serves it except vLLM, whose token_embed pooling type plus late-interaction MaxSim scoring is the only first-class ColBERT endpoint in the repo [[vllm-pooling-models]]. TEI cannot: its pooling options are cls, mean, splade and last-token, with no token-wise output mode, so 'a ColBERT model cannot be served correctly here' [[hf-text-embeddings-inference]]. llama.cpp-embedding-server cannot either: --pooling none gives raw per-token output, 'which is a building block for ColBERT but not a served ColBERT' [[llama-cpp-embedding-server]]. BATCHING TRAP, cited: reranking through Triton's sequence batcher requires modelling the query-document pair as ONE stateful unit rather than two independent sequences [[triton-inference-server]] - get that wrong and the pair is scored separately.