GTE-Qwen2-1.5B-instruct

KV CACHE CORRECTED 2026-10-04 (parent agent): was 14336.0, which is wrong by 2x. Correct value 28672 = 28 layers * 2 kv_heads * 128 head_dim * 2 (K+V) * 2 (bf16). The record's own note showed the formula but its arithmetic did not match it. THE EXCEPTION in the embedding slice: this embedder is a CAUSAL DECODER, so unlike every other record here it DOES have a KV cache by construction, and the formula applies. head_dim derived = 1536 / 12 = 128. kv_cache_bytes_per_token = 28 * 2 * 128 * 2 * 2 = 14336 bytes/token at bf16 — small, because Qwen2 GQA here is 12 query / 2 KV heads. But for embedding use it is IRRELEVANT: embedding is a single forward pass with last-token pooling, the cache is never grown, so resident memory is dominated by the 1.78B weights, not by KV. Recording the number anyway because the architecture is a decoder and a serving stack that sizes memory by the KV formula will get a wrong answer for this model. Serving profile: 32x the compute of a BERT-base embedder (3.55e9 vs 2.19e8 flops/token) and 5x the parameters of bge-large. It exists because it scores materially better on retrieval benchmarks and handles 131072 max_position_embeddings natively (sliding_window 131072, effectively disabled, rope_theta 1e6). The cost is that it is GPU-class work — unlike bge-small, this one is not faster on CPU. Vocab 151646 dominates the parameter count: 151646 * 1536 = 233M of the 1.776B is the embedding table. Apache-2.0. GGUF ports exist (second-state, gaianet) and ONNX (AnotherWallace/keisuke-miyako int8, twright8 fp16) — the int8 GGUF and ONNX builds are the practical CPU story for this class. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=bi-encoder. REQUIRED KERNEL CLASS: a single full-sequence bidirectional forward pass with a pooling reduction. No KV cache, no incremental state, no autoregressive weight streaming - flop/embedding-batch-encoding gives AI = B*t flop/byte (weights read once for all B documents), so on H100 SXM at ridge 295.4 a single 128-token encode at B=1 is memory-bound but at B=3 it is compute-bound, and the threshold is the PRODUCT B*t. flop/cross-encoder-reranking makes the same point for the scoring phase. This inverts the hardware-selection rule: an embedding/rerank tier wants FLOPS and core count, not HBM bandwidth. SUPPORTED BY (cited): vLLM pooling is a separate runner with its own endpoint surface and the broadest architecture table in the repo - BertModel, BgeM3EmbeddingModel, GteModel, JteNewModel (mGTE-TRM), JinaEmbeddingsV5Model, LlamaBidirectionalModel, LlamaModel/MistralModel (intfloat/e5-mistral-7b-instruct), ModernBertModel, NomicBertModel, Qwen2Model, Qwen3Model, RobertaModel, XLMRobertaModel, plus any generative model via automatic --convert, with the Transformers modeling backend covering encoder-only architectures directly [[vllm-pooling-models]]. TEI is the purpose-built reference and lists BERT, CamemBERT, XLM-RoBERTa, NomicBERT, JinaBERT (Alibi), MPNet, ModernBERT plus RoPE decoder backbones Mistral, Alibaba GTE, Qwen2, Qwen3 and Gemma3 [[hf-text-embeddings-inference]]. SGLang's embedding page states native encoder architectures are auto-detected while decoder-style embedders need --is-embedding, and that on CUDA it automatically uses breakable CUDA graph for the full encoder prefill and DISABLES radix-cache and chunked-prefill [[mdl2-sglang-embedding-rerank-models]]. sentence-transformers covers everything transformers can load [[sentence-transformers]]. llama.cpp reaches BERT-family encoders via the LLM_ARCH_BERT / MODERN_BERT / NOMIC_BERT / JINA_BERT_V2 / JINA_BERT_V3 entries [[pap-arch-llamacpp-arch-registry]]. CORRECTNESS CAVEATS THAT ARE ARCHITECTURE-SPECIFIC AND CITED: vLLM's pooling path is a capability not a speed win - the project's own docs say 'We currently support pooling models primarily for convenience. This is not guaranteed to provide any performance improvements over using Hugging Face Transformers or Sentence Transformers directly' [[vllm-pooling-models]]. Several of the architectures above need --trust-remote-code or --hf-overrides, so 'supported' sometimes means 'supported after three flags' [[vllm-pooling-models]]. bge-m3 ships extra sparse AND ColBERT weights with a separate docs path, and jina-embeddings-v3 is limited to text-matching with LoRA merged rather than selected [[vllm-pooling-models]]. TEI's --dtype accepts only float16 and float32 - no int8 or fp8 at runtime, so quantization must happen at export [[hf-text-embeddings-inference]]. llama.cpp's embedding path cannot serve SPLADE/learned-sparse encoders, cannot serve ColBERT end to end, and cannot serve multimodal encoders requiring a projector graph [[llama-cpp-embedding-server]]. For the Qwen2-decoder-as-bi-encoder pattern specifically, both gte-Qwen2 and Qwen2 family models are the documented --trust-remote-code case in vLLM [[vllm-pooling-models]].