Models (191)

Released models and what their architecture does to the FLOP mix. kv_cache_bytes_per_token is the number that decides whether a model fits at all.

draft 15verified 176→ sources (2562)

Filter

architecture

family

vendor

context scaling method

status

sort

Based-1.3B

hybrid-attention-ssm 1.3B50,176 B/token KVthree-way interleave: short-conv (BaseConv) on most layers + Taylor linear attention on 7 layers + sliding-window MHA (window 128) on 7 layers2,048 ctxmodels/based-1-3b

draftlow · 0.50

Claude Haiku 4.5

other200,000 ctxmodels/claude-haiku-4-5

draftlow · 0.30

Claude Opus 5.5

other1,000,000 ctxmodels/claude-opus-5-5

draftlow · 0.30

Claude Sonnet 5.5

other1,000,000 ctxmodels/claude-sonnet-5-5

draftlow · 0.30

GPT-4.1

other1,047,576 ctxmodels/gpt-4-1

draftlow · 0.35

Hyena

recurrentimplicit long convolution + data-controlled gatingmodels/hyena-arch

draftlow · 0.50

Mercury

diffusion0 B/token KVdiffusion (non-autoregressive)models/mercury

draftlow · 0.40

other200,000 ctxmodels/o3

draftlow · 0.30

RetNet

recurrentretention (multi-scale exponential decay)models/retnet-arch

draftlow · 0.50

Samba-7B

hybrid-attention-ssm 3.8Bhybrid: layer-wise Mamba + sliding-window attention (w=2048)models/samba-7b

draftlow · 0.60

Voyage-3

densemodels/voyage-3

draftlow · 0.30

Voyage-3-Lite

densemodels/voyage-3-lite

draftlow · 0.30

Voyage-Code-3

densemodels/voyage-code-3

draftlow · 0.40

Whisper large-v1

dense 1.55B163,840 B/token KVMHA, encoder-decoder (cross-attention)1,500 ctxmodels/whisper-large-v1

draftlow · 0.50

Zamba-7B

hybrid-attention-ssm 7B193,024 B/token KVhybrid: Mamba-2 backbone with a SINGLE SHARED attention block, invoked at 13 of 76 layers; MHA not GQA (16 query heads, 16 KV heads)4,096 ctxmodels/zamba-7b

draftlow · 0.60

AliceAI-Foundation-80B-A3B-Base

hybrid-attention-ssm 81.286B24,576 B/token KVhybrid: 36 of 48 layers are KDA linear attention, 12 are gated full attention (every 4th layer, indices 3,7,...,47); GQA 16/2262,144 ctxmodels/aliceai-foundation-80b-a3b-base

verifiedmedium · 0.80

all-MiniLM-L6-v2

dense 0.0227B0 B/token KVbidirectional encoder (BERT), full attention, no causal mask512 ctxmodels/all-minilm-l6-v2

verifiedhigh · 0.95

all-mpnet-base-v2

dense 0.109B0 B/token KVbidirectional encoder (MPNet), full attention, no causal mask, relative position bias514 ctxmodels/all-mpnet-base-v2

verifiedhigh · 0.95

Beam 1.0 501B-A23B

moe 23.4/501.3B active36,864 B/token KVGated GQA sliding-window hybrid: 43 of 52 layers are sliding_attention (window 512, 8 kv heads, head_dim 128) and 9 are full_attention, at indices 5, 11, 17, 23, 29, 35, 41, 47, 511,048,576 ctxmodels/beam-1-0-501b-a23b

verifiedmedium · 0.80

Bespoke-Stratos-7B

dense 7.61562B57,344 B/token KVGQA32,768 ctxmodels/bespoke-stratos-7b

verifiedhigh · 0.88

BGE Base English v1.5

dense 0.109B0 B/token KVbidirectional encoder (BERT), full attention, no causal mask512 ctxmodels/bge-base-en-v1-5

verifiedhigh · 0.95

BGE Large English v1.5

dense 0.335B0 B/token KVbidirectional encoder (BERT), full attention, no causal mask512 ctxmodels/bge-large-en-v1-5

verifiedhigh · 0.95

BGE Reranker Large

dense 0.578B0 B/token KVcross-encoder (XLM-RoBERTa), full bidirectional attention over query+passage514 ctxmodels/bge-reranker-large

verifiedhigh · 0.90

BGE Reranker v2 M3

dense 0.568B0 B/token KVcross-encoder (XLM-RoBERTa), full bidirectional attention over query+passage8,194 ctxmodels/bge-reranker-v2-m3

verifiedhigh · 0.90

BGE Small English v1.5

dense 0.0334B0 B/token KVbidirectional encoder (BERT), full attention, no causal mask512 ctxmodels/bge-small-en-v1-5

verifiedhigh · 0.95

BGE-M3

dense 0.568B0 B/token KVbidirectional encoder (XLM-RoBERTa), full attention, no causal mask8,194 ctxmodels/bge-m3

verifiedhigh · 0.85

Canary-1B-Flash (encoder checkpoint)

dense 0.811B0 B/token KVMHA (no GQA); FastConformer encoder only5,000 ctxmodels/canary-1b-flash

verifiedhigh · 0.90

Cohere Embed English v3.0

dense0 B/token KVbidirectional encoder (architecture not fully specified in config)512 ctxmodels/cohere-embed-english-v3

verifiedhigh · 0.85

Cohere Embed Multilingual v3.0

dense0 B/token KVbidirectional encoder (architecture not fully specified in config)512 ctxmodels/cohere-embed-multilingual-v3

verifiedhigh · 0.85

Cohere Rerank v3.5

dense0 B/token KVcross-encoder (architecture not specified)4,096 ctxmodels/cohere-rerank-v3-5

verifiedmedium · 0.70

Darwin-180B-RSI

hybrid-attention-ssm 180B24,576 B/token KVhybrid: linear attention with full attention every 4th layer (12 of 48)262,144 ctxmodels/darwin-180b-rsi

verifiedmedium · 0.70

DeepSeek-R1

moe 37/671B active70,272 B/token KVMLA, kv_lora_rank 512, q_lora_rank 1536 (identical to DeepSeek-V3)163,840 ctxMLA latent 512models/deepseek-r1

verifiedhigh · 0.90

DeepSeek-R1-Distill-Llama-70B

dense 70.5537B327,680 B/token KVGQA131,072 ctxmodels/deepseek-r1-distill-llama-70b

verifiedhigh · 0.93

DeepSeek-R1-Distill-Llama-8B

dense 8.03026B131,072 B/token KVGQA131,072 ctxmodels/deepseek-r1-distill-llama-8b

verifiedhigh · 0.92

DeepSeek-R1-Distill-Qwen-1.5B

dense 1.77709B28,672 B/token KVGQA131,072 ctxmodels/deepseek-r1-distill-qwen-1p5b

verifiedhigh · 0.92

DeepSeek-R1-Distill-Qwen-32B

dense 32.8B262,144 B/token KVGQA (standard; sliding_window set to full 131072 = inactive)131,072 ctxmodels/deepseek-r1-distill-qwen-32b

verifiedhigh · 0.90

DeepSeek-R1-Distill-Qwen-7B

dense 7.61562B57,344 B/token KVGQA131,072 ctxmodels/deepseek-r1-distill-qwen-7b

verifiedhigh · 0.92

DeepSeek-V2

moe 21/236B active69,120 B/token KVMLA, kv_lora_rank 512, q_lora_rank 1536163,840 ctxMLA latent 512models/deepseek-v2

verifiedhigh · 0.90

DeepSeek-V3

moe 37/671B active70,272 B/token KVMLA (multi-head latent attention), q_lora_rank 1536, kv_lora_rank 512, 128 heads expanded from latent163,840 ctxMLA latent 512models/deepseek-v3

verifiedhigh · 0.90

DeepSeek-V3.2

moe 37/685.4B active70,272 B/token KVMLA + sparse-attention indexer (index_topk 2048)163,840 ctxMLA latent 512models/deepseek-v3-2

verifiedhigh · 0.90

DeepSeek-V4-Flash

moe 13/284B activeMLA variant with compressed/hashed KV (sliding_window 128, compress_ratios), sparse indexer index_topk 512; kv_lora_rank absent from config1,048,576 ctxmodels/deepseek-v4-flash

verifiedhigh · 0.90

DeepSeek-V4-Pro

moe 49/1600B activeMLA variant with compressed KV, sliding_window 128, sparse indexer index_topk 1024; kv_lora_rank absent from config1,048,576 ctxmodels/deepseek-v4-pro

verifiedhigh · 0.90

DeepSeek-V4-Pro-0813

moe 49/1600B activeMLA variant with compressed KV, sliding_window 128, sparse indexer index_topk 1024; kv_lora_rank absent from config1,048,576 ctxmodels/deepseek-v4-pro-0813

verifiedmedium · 0.84

DeepSeek-V4.1-Flash-1017

moe 8/552B active890 B/token KVCSA2 (Full/Reindex/Reuse) over a Causal Encoder-Decoder; FP4 main KV; no kv_lora_rank, so not V3-style MLA1,048,576 ctxmodels/deepseek-v4-1-flash-1017

verifiedmedium · 0.80

Dolphin 2.9 Llama3 8B

dense 8.03028B131,072 B/token KVGQA8,192 ctxmodels/dolphin-2-9-llama3-8b

verifiedhigh · 0.90

E5 Base v2

dense 0.109B0 B/token KVbidirectional encoder (BERT), full attention, no causal mask512 ctxmodels/e5-base-v2

verifiedhigh · 0.95

E5 Large v2

dense 0.335B0 B/token KVbidirectional encoder (BERT), full attention, no causal mask512 ctxmodels/e5-large-v2

verifiedhigh · 0.90

E5 Mistral 7B Instruct

dense 7B131,072 B/token KVcausal decoder (Mistral) used as a bi-encoder via last-token pooling32,768 ctxmodels/e5-mistral-7b-instruct

verifiedhigh · 0.85

Falcon-H1-7B-Base

hybrid-attention-ssm 7.58565B45,056 B/token KVGQA in parallel with Mamba-2 on every layer262,144 ctxmodels/falcon-h1-7b-base

verifiedhigh · 0.88

Falcon3 7B Instruct

dense 7.45555B114,688 B/token KVGQA32,768 ctxmodels/falcon3-7b-instruct

verifiedhigh · 0.92

Gemma 1 7B

dense 8.538B458,752 B/token KVMHA (no softcapping, no sliding window)8,192 ctxmodels/gemma-1-7b

verifiedhigh · 0.85

Gemma 2 2B

dense 2.614B106,496 B/token KVGQA + logit softcapping (attn 50.0, final 30.0) + sliding-window (4096, 5:1 local:global)8,192 ctxmodels/gemma-2-2b

verifiedhigh · 0.90

Gemma 2 9B

dense 9.24B344,064 B/token KVGQA + alternating sliding-window/global (sliding_window 4096)8,192 ctxmodels/gemma-2-9b

verifiedhigh · 0.90

Gemma 3 12B IT (multimodal)

dense 12.187B393,216 B/token KVsliding-window/hybrid (5 local : 1 global)131,072 ctxmodels/gemma-3-12b-it

verifiedhigh · 0.90

Gemma 3 27B (multimodal)

dense 27.43B507,904 B/token KVGQA + 5:1 sliding-window/global (sliding_window 1024, pattern 6)131,072 ctxmodels/gemma-3-27b

verifiedhigh · 0.90

Gemma 3 4B IT (multimodal)

dense 4.3B139,264 B/token KVsliding-window/hybrid (5 local : 1 global)131,072 ctxmodels/gemma-3-4b-it

verifiedhigh · 0.90

Gemma 3n E2B

dense 5.44B61,440 B/token KVGQA + 4:1 sliding-window/global (sliding_window 512)32,768 ctxmodels/gemma-3n-e2b

verifiedmedium · 0.80

Gemma 4 12B (unified multimodal)

dense 11.96B344,064 B/token KVGQA + sliding-window/global hybrid with separate global head geometry (global_head_dim 512, num_global_key_value_heads 1)262,144 ctxmodels/gemma-4-12b

verifiedmedium · 0.80

Gemma 4 26B A4B (MoE, multimodal)

moe 4/25.81B active225,280 B/token KVGQA + sliding-window/global hybrid with separate global head geometry (global_head_dim 512, num_global_key_value_heads 2)262,144 ctxmodels/gemma-4-26b-a4b

verifiedhigh · 0.85

Gemma 4 31B (dense, multimodal)

dense 31.27B901,120 B/token KVGQA + sliding-window/global hybrid with separate global head geometry (global_head_dim 512, num_global_key_value_heads 4)262,144 ctxmodels/gemma-4-31b

verifiedhigh · 0.85

GLM-4.1V-9B-Thinking

dense 10.293B40,960 B/token KVGQA (MQA-like, 2 KV heads) + mrope65,536 ctxmodels/glm-4-1v-9b-thinking

verifiedhigh · 0.90

GLM-4.5

moe 32/355B active376,832 B/token KVGQA (standard) - NOT MLA despite the DeepSeek-derived MoE design131,072 ctxmodels/glm-4-5

verifiedhigh · 0.90

GLM-4.7-Flash

moe 3/31.2B active54,144 B/token KVMLA (kv_lora_rank 512, qk_nope 192 + qk_rope 64, v_head_dim 256)202,752 ctxMLA latent 512models/glm-4-7-flash

verifiedhigh · 0.90

GLM-5.3

moe 40/753.3B active89,856 B/token KVMLA + DSA sparse attention indexer (index_topk 2048), kv_lora_rank 5121,048,576 ctxMLA latent 512models/glm-5-3

verifiedhigh · 0.90

GLM-5.3-Flash

moe 18/320B active11,264 B/token KVhybrid: 34 KDA linear-attention layers + 11 MLA/DSA sparse-attention layers (every 4th)1,048,576 ctxMLA latent 512models/glm-5-3-flash

verifiedhigh · 0.90

GPT-OSS 120B

moe 5.1/116.8B active73,728 B/token KVGQA + sliding-window/banded hybrid131,072 ctxmodels/gpt-oss-120b

verifiedhigh · 0.95

GPT-OSS 20B

moe 3.6/20.9B active49,152 B/token KVGQA + sliding-window/banded hybrid131,072 ctxmodels/gpt-oss-20b

verifiedhigh · 0.95

Granite 3.3 8B Instruct

dense 8.17086B163,840 B/token KVGQA131,072 ctxmodels/granite-3-3-8b-instruct

verifiedhigh · 0.85

Granite 4.0 H Tiny

hybrid-attention-ssm 6.93904B8,192 B/token KVGQA on 4 of 40 layers; Mamba-2 on 36; NoPE131,072 ctxmodels/granite-4-0-h-tiny

verifiedhigh · 0.86

Griffin-14B

hybrid-attention-ssm 14B0 B/token KVhybrid: RG-LRU gated linear recurrence plus local sliding-window MQA, 2 recurrent residual blocks then 1 local-attention block, window 1024models/griffin-14b

verifiedmedium · 0.80

Grok 1

moe 314B262,144 B/token KVGQA8,192 ctxmodels/grok-1

verifiedhigh · 0.85

GTE Base English v1.5

dense 0.109B0 B/token KVbidirectional encoder (NewModel), full attention, no causal mask8,192 ctxmodels/gte-base

verifiedhigh · 0.90

GTE Large

dense 0.335B0 B/token KVbidirectional encoder (BERT), full attention, no causal mask512 ctxmodels/gte-large

verifiedhigh · 0.95

GTE Reranker ModernBERT

dense 0.149B0 B/token KVcross-encoder (ModernBERT), full bidirectional attention over query+passage with local+global attention pattern8,192 ctxmodels/gte-reranker-modernbert

verifiedhigh · 0.85

GTE-Qwen2-1.5B-instruct

dense 1.776B28,672 B/token KVcausal decoder (Qwen2) used as a bi-encoder via last-token pooling131,072 ctxmodels/gte-qwen2-1-5b-instruct

verifiedhigh · 0.85

Hermes-3-Llama-3.1-8B

dense 8.03026B131,072 B/token KVGQA131,072 ctxmodels/hermes-3-llama-3-1-8b

verifiedhigh · 0.93

Hermes-4-405B

dense 405.853B516,096 B/token KVGQA131,072 ctxmodels/hermes-4-405b

verifiedhigh · 0.93

Hunyuan-A13B-Instruct

moe 13/80B active131,072 B/token KVGQA (standard; no MLA - kv_lora_rank explicitly null)32,768 ctxmodels/hunyuan-a13b

verifiedhigh · 0.85

Hy-Memory-30B-A3B

moe 3/30.06B active98,304 B/token KVGQA (32 query heads / 4 kv heads, head_dim 128); dense first layer then MoE262,144 ctxmodels/hy-memory-30b-a3b

verifiedhigh · 0.86

Intern-S2-397B

moe 17/403.423B active15,360 B/token KVGQA plus gated linear attention: 45 of 60 layers are linear_attention and 15 are full_attention, one every fourth layer262,144 ctxmodels/intern-s2-397b

verifiedmedium · 0.80

InternVL2_5-8B

dense 8.075B131,072 B/token KVGQA32,768 ctxmodels/internvl2-5-8b

verifiedhigh · 0.90

InternVL3 14B

dense 14B196,608 B/token KVGQA in the LLM tower (Qwen2ForCausalLM) plus a separate ViT encoder; no attention interaction between the two at serving time32,768 ctxmodels/internvl3-14b

verifiedmedium · 0.82

InternVL3-8B

dense 7.944B57,344 B/token KVGQA32,768 ctxmodels/internvl3-8b

verifiedhigh · 0.90

Jamba-1.5-Large

hybrid-attention-ssm36,864 B/token KVhybrid: 9 attention layers (every 8th, starting at layer 4) with 64 query / 8 KV heads, head_dim 128; the other 63 layers are Mamba (state 16, conv 4) and carry no KV262,144 ctxmodels/jamba-1-5-large

verifiedhigh · 0.86

Jina Embeddings v3

dense 0.572B0 B/token KVbidirectional encoder (XLM-RoBERTa with LoRA adapters), full attention, no causal mask8,194 ctxmodels/jina-embeddings-v3

verifiedhigh · 0.90

Jina Embeddings v4

dense 3.24B36,864 B/token KVcausal decoder (Qwen2.5-VL text backbone) used as a bi-encoder via mean pooling, with multi-vector projector128,000 ctxmodels/jina-embeddings-v4

verifiedhigh · 0.85

Jina Reranker v2 Base Multilingual

dense 0.278B0 B/token KVcross-encoder (XLM-RoBERTa), full bidirectional attention over query+passage1,026 ctxmodels/jina-reranker-v2-base-multilingual

verifiedhigh · 0.90

K2-Horizon-MoVA-36B-A4B

moe 4/37.445B active196,608 B/token KVGQA plus Mixture-of-Values attention (MoVA): 64 value experts, 4 routed per token524,288 ctxmodels/k2-horizon-mova-36b-a4b

verifiedmedium · 0.70

Kimi-Audio-7B-Instruct

dense 9.766B57,344 B/token KVGQA8,192 ctxmodels/kimi-audio-7b-instruct

verifiedhigh · 0.90

Kimi-K2

moe 32/1026B active70,272 B/token KVMLA, kv_lora_rank 512, q_lora_rank 1536131,072 ctxMLA latent 512models/kimi-k2

verifiedhigh · 0.90

Kimi-K2.6

moe 32/1026.88B active70,272 B/token KVMLA, kv_lora_rank 512, q_lora_rank 1536262,144 ctxmodels/kimi-k2-6

verifiedhigh · 0.86

Kimi-K3

hybrid-attention-ssm 2780B27,648 B/token KVhybrid: Gated MLA on 24 layers + Kimi Delta Attention on 69 layers, with Attention Residuals1,048,576 ctxMLA latent 512models/kimi-k3

verifiedhigh · 0.90

Kimi-Linear-48B-A3B-Instruct

hybrid-attention-ssm 48B8,064 B/token KVhybrid: MLA on 7 full-attention layers + Kimi Delta Attention (linear) on 20 layers1,048,576 ctxMLA latent 512models/kimi-linear-48b-a3b

verifiedhigh · 0.90

Kolibri-1

moe 3.458/78.103B active20,480 B/token KVhybrid sliding-window: 40 of 50 layers are sliding_attention with window 513, 10 are full_attention (every 5th layer, indices 4,9,...,49); GQA 48/4262,144 ctxmodels/kolibri-1

verifiedmedium · 0.72

LFM2-1.2B

hybrid-attention-ssm 1.17B0 B/token KVhybrid: 10 short-conv (Mamba-style) + 6 full-attention layers128,000 ctxmodels/lfm2-1-2b

verifiedhigh · 0.90

LFM2.5-1.2B-Instruct

hybrid-attention-ssm 1.17B0 B/token KVhybrid: layer_types = 10x conv + 6x full_attention of 16128,000 ctxmodels/lfm2-5-1-2b-instruct

verifiedhigh · 0.90

LFM2.5-24B-A2B

moe 2.45/24B active20,480 B/token KVhybrid: 30 double-gated short-convolution blocks + 10 GQA full-attention layers, top-4 MoE131,072 ctxmodels/lfm2-5-24b-a2b

verifiedhigh · 0.86

Ling-3.0-flash

hybrid-attention-ssm 124B8,064 B/token KVhybrid: MLA (kv_lora_rank 512) + Kimi Delta Attention, 5:1 alternating, 20 window layers262,144 ctxMLA latent 512models/ling-3-0-flash

verifiedhigh · 0.85

Ling-3.1-flash

hybrid-attention-ssm 560B10,368 B/token KVhybrid KDA + gated MLA: 63 of 72 layers are Kimi Delta Attention (fixed recurrent state) and 9 are gated MLA262,144 ctxMLA latent 512models/ling-3-1-flash

verifiedmedium · 0.80

LLaDA-8B-Instruct

diffusion 8.01558B0 B/token KVfull bidirectional attention, no causal mask4,096 ctxmodels/llada-8b-instruct

verifiedhigh · 0.85

LLaDA2.2-mini

diffusion 16.256B0 B/token KVbidirectional diffusion attention (use_cache false); GQA 16/4; partial RoPE factor 0.5; MoE with one shared expert131,072 ctxmodels/llada2-2-mini

verifiedmedium · 0.84

Llama 3.1 8B

dense 8.03B131,072 B/token KVGQA131,072 ctxmodels/llama-3-1-8b

verifiedhigh · 0.90

Llama 3.3 70B

dense 70.55B327,680 B/token KVGQA131,072 ctxmodels/llama-3-3-70b

verifiedhigh · 0.90

Llama 4 Maverick (17B active / 400B total)

moe 17/401.6B active196,608 B/token KVGQA1,048,576 ctxmodels/llama-4-maverick

verifiedhigh · 0.85

Llama 4 Scout (17B active / 109B total)

moe 17/108.6B active196,608 B/token KVGQA10,485,760 ctxmodels/llama-4-scout

verifiedhigh · 0.85

LLaVA-OneVision-Qwen2-7B (HF format)

dense 8.031B57,344 B/token KVGQA (LLM); SigLIP full attention (vision)32,768 ctxmodels/llava-onevision-qwen2-7b

verifiedhigh · 0.90

Magistral Small

dense 24.01B163,840 B/token KVGQA131,072 ctxmodels/magistral-small

verifiedmedium · 0.80

Mamba-Codestral-7B-v0.1

ssm 7.2854B0 B/token KVnone - pure Mamba-2, no attention layers at allmodels/mamba-codestral-7b

verifiedhigh · 0.90

Mellum2.1 Thinking 12B A2.5B

moe 2.5/12.15B active7,168 B/token KVGQA, hybrid sliding-window: 21 of 28 layers are sliding_attention (window 1024) and 7 are full_attention, one every fourth layer131,072 ctxmodels/mellum2-1-12b-a2-5b-thinking

verifiedhigh · 0.86

MiMo-V2.6-Flash-RL

moe 15/309B active23,040 B/token KVGQA sliding-window hybrid: 39 of 48 layers are sliding-window attention (window 128, 8 kv heads) and 9 are global (4 kv heads)1,048,576 ctxmodels/mimo-v2-6-flash-rl

verifiedmedium · 0.82

MiMo-V2.6-Pro-RL

moe 42/1000B active51,200 B/token KVGQA sliding-window hybrid: 60 of 70 layers are sliding-window attention (window 128, 8 kv heads) and 10 are global (8 kv heads)1,048,576 ctxmodels/mimo-v2-6-pro-rl

verifiedmedium · 0.75

MiniCPM-o-2_6 (omni: vision + audio + speech)

dense 8.675B57,344 B/token KVGQA32,768 ctxmodels/minicpm-o-2-6

verifiedhigh · 0.90

MiniCPM-V 4.7 35B-A3B

hybrid-attention-ssm 35.213B20,480 B/token KVhybrid: linear attention with full attention every 4th layer (10 of 40); partial RoPE; shared-expert MoE262,144 ctxmodels/minicpm-v-4-7-35b-a3b

verifiedhigh · 0.85

MiniCPM-V-4_5

dense 8.696B147,456 B/token KVGQA40,960 ctxmodels/minicpm-v-4-5

verifiedhigh · 0.90

MiniCPM-V-4_6

hybrid-attention-ssm 1.3B12,288 B/token KVhybrid linear-attention/full-attention (18:6) + gated attention262,144 ctxmodels/minicpm-v-4-6

verifiedhigh · 0.90

MiniCPM5-2B

dense 2.517B43,008 B/token KVGQA, 16 query heads and 2 kv heads131,072 ctxmodels/minicpm5-2b

verifiedmedium · 0.84

MiniMax-M1

hybrid-attention-ssm 456B40,960 B/token KVhybrid: lightning (linear) attention on 70 layers + full GQA on 10 layers10,240,000 ctxmodels/minimax-m1

verifiedhigh · 0.85

MiniMax-M2.7

moe 10/229B active253,952 B/token KVGQA, all-full attention (attn_type_list all 1; M2 line abandoned M1's hybrid lightning attention)204,800 ctxmodels/minimax-m2-7

verifiedhigh · 0.85

Ministral 3 8B (multimodal)

dense 8.92B139,264 B/token KVGQA262,144 ctxmodels/ministral-3-8b

verifiedhigh · 0.90

Ministral 8B

dense 8.02B147,456 B/token KVGQA32,768 ctxmodels/ministral-8b

verifiedhigh · 0.90

Mixedbread mxbai-embed-large-v1

dense 0.335B0 B/token KVbidirectional encoder (BERT), full attention, no causal mask512 ctxmodels/mxbai-embed-large-v1

verifiedhigh · 0.90

Mixedbread mxbai-rerank-large-v1

dense 0.435B0 B/token KVcross-encoder (DeBERTa-v2), full bidirectional attention over query+passage512 ctxmodels/mxbai-rerank-large-v1

verifiedhigh · 0.90

Mixedbread mxbai-rerank-large-v2

dense 1.776B28,672 B/token KVcross-encoder (Qwen2 causal decoder), full attention over query+passage32,768 ctxmodels/mxbai-rerank-large-v2

verifiedhigh · 0.85

Mixtral 8x22B

moe 39/140.6B active229,376 B/token KVGQA65,536 ctxmodels/mixtral-8x22b

verifiedhigh · 0.90

Mixtral 8x7B

moe 13/46.7B active131,072 B/token KVGQA32,768 ctxmodels/mixtral-8x7b

verifiedhigh · 0.95

ModernBERT Base

dense 0.15B0 B/token KVbidirectional encoder, alternating local (128) and global attention (every 3rd layer)8,192 ctxmodels/modernbert-base

verifiedhigh · 0.90

Molmo-7B-D-0924

dense 8.021B57,344 B/token KVGQA4,096 ctxmodels/molmo-7b-d

verifiedhigh · 0.90

Naive-N0.5-Flash

moe 15.5/309B active23,040 B/token KVGQA hybrid with no full attention: 39 sliding-window layers (window 128, 8 kv heads) and 9 DeepSeek Sparse Attention layers (4 kv heads, indexer top-2048)1,048,576 ctxmodels/naive-n0-5-flash

verifiedmedium · 0.80

Nemotron-H-8B-Base-8K

hybrid-attention-ssm 8.10085B16,384 B/token KVGQA on 4 of 52 layers; Mamba-2 on 488,192 ctxmodels/nemotron-h-8b-base-8k

verifiedhigh · 0.90

Nomic Embed Text v1.5

dense 0.137B0 B/token KVbidirectional encoder (nomic_bert: BERT + rotary, GLU MLP)2,048 ctxmodels/nomic-embed-text-v1-5

verifiedhigh · 0.90

Nomic Embed Text v2 MoE

moe 0.058/0.494B active0 B/token KVbidirectional encoder (nomic_bert with MoE), full attention, no causal mask, MHA (12 heads, gqa_ratio 1.0)2,048 ctxmodels/nomic-embed-text-v2-moe

verifiedhigh · 0.85

Nous-Hermes-2-Mixtral-8x7B-DPO

moe 12.9/46.7028B active131,072 B/token KVGQA32,768 ctxmodels/nous-hermes-2-mixtral-8x7b-dpo

verifiedhigh · 0.87

NVIDIA Nemotron 3.5 Bolt 74B-A9B

hybrid-attention-ssm 74B8,192 B/token KVhybrid Mamba-2 + LatentMoE + GQA: 40 Mamba-2, 40 MoE, 8 attention (of 88 blocks); vision via C-RADIO v4-H262,144 ctxmodels/nemotron-3-5-bolt-74b-a9b

verifiedmedium · 0.84

NVIDIA Nemotron 3.5 Super VL 120B-A12B

hybrid-attention-ssm 120B8,192 B/token KVhybrid Mamba-2 + LatentMoE + GQA: 40 Mamba-2, 40 MoE, 8 attention (of 88 blocks); vision via C-RADIO v4-H1,048,576 ctxmodels/nemotron-3-5-super-vl-120b-a12b

verifiedmedium · 0.84

NVIDIA Nemotron Labs Dory 1.0 4B

recurrent 3.975B147,456 B/token KVlooped nGPT: interleaved sliding-window (1024) and full GQA; shared recurrent block applied n_recurrent_loops times (default 2) with input reinjection262,144 ctxmodels/nemotron-labs-dory1-4b

verifiedmedium · 0.84

OLMo 2 1B

dense 1.485B131,072 B/token KVMHA (no GQA)4,096 ctxmodels/olmo-2-1b

verifiedhigh · 0.93

Olmo 3 7B

dense 7.3B524,288 B/token KVfull MHA (no GQA) + 3:1 sliding-window/global (sliding_window 4096)65,536 ctxmodels/olmo-3-7b

verifiedhigh · 0.90

OpenHermes-2.5-Mistral-7B

dense 7.24175B131,072 B/token KVGQA32,768 ctxmodels/openhermes-2-5-mistral-7b

verifiedhigh · 0.90

OpenR1-Qwen-7B

dense 7.61562B57,344 B/token KVGQA32,768 ctxmodels/openr1-qwen-7b

verifiedhigh · 0.85

OpenThinker3-7B

dense 7.61562B57,344 B/token KVGQA32,768 ctxmodels/openthinker3-7b

verifiedhigh · 0.88

Ornith-1.5-35B-A3B

moe 3/35.952B active20,480 B/token KVGQA plus gated linear attention: 30 of 40 layers are linear_attention and 10 are full_attention, one every fourth layer262,144 ctxmodels/ornith-1-5-35b-a3b

verifiedmedium · 0.80

Ornith-1.5-9B

dense 9.653B32,768 B/token KVGQA plus gated linear attention: 24 of 32 layers are linear_attention and 8 are full_attention, one every fourth layer262,144 ctxmodels/ornith-1-5-9b

verifiedmedium · 0.80

PaliGemma 3B mix 224

dense 2.923B18,432 B/token KVMQA: 8 query heads, 1 KV head, head_dim 256 (Gemma-2B text tower)8,192 ctxmodels/paligemma-3b-mix-224

verifiedhigh · 0.88

Parakeet CTC 1.1B

dense 1.063B0 B/token KVMHA (no GQA); FastConformer encoder only5,000 ctxmodels/parakeet-ctc-1-1b

verifiedhigh · 0.90

Parakeet TDT 0.6B v2

dense 0.6Bmodels/parakeet-tdt-0-6b-v2

verifiedhigh · 0.90

Phi-3-mini-4k-instruct

dense 3.821B393,216 B/token KVMHA + sliding-window (2047)4,096 ctxmodels/phi-3-mini-4k-instruct

verifiedhigh · 0.95

Phi-3.5-mini-instruct

dense 3.821B393,216 B/token KVMHA + sliding-window (262144, effectively disabled)131,072 ctxmodels/phi-3-5-mini-instruct

verifiedhigh · 0.95

Phi-3.5-vision-instruct

dense 4.147B393,216 B/token KVMHA (no GQA)131,072 ctxmodels/phi-3-5-vision-instruct

verifiedhigh · 0.90

Phi-4-mini-instruct

dense 3.836B131,072 B/token KVGQA + partial RoPE (0.75) + sliding-window (262144)131,072 ctxmodels/phi-4-mini-instruct

verifiedhigh · 0.95

Phi-4-mini-reasoning

dense 3.836B131,072 B/token KVGQA + partial RoPE (0.75) + sliding-window (262144)131,072 ctxmodels/phi-4-mini-reasoning

verifiedhigh · 0.93

Phi-4-multimodal-instruct

dense 5.574B131,072 B/token KVGQA + partial RoPE (0.75)131,072 ctxmodels/phi-4-multimodal-instruct

verifiedhigh · 0.90

Pixtral-12B-2409

dense 12B163,840 B/token KVGQA131,072 ctxmodels/pixtral-12b-2409

verifiedhigh · 0.90

Qwen2-Audio-7B-Instruct

dense 8.397B524,288 B/token KVMHA (no GQA)8,192 ctxmodels/qwen2-audio-7b-instruct

verifiedhigh · 0.90

Qwen2-VL-7B-Instruct

dense 8.291B57,344 B/token KVGQA + mrope32,768 ctxmodels/qwen2-vl-7b-instruct

verifiedhigh · 0.90

Qwen2.5-72B-Instruct

dense 72.7B327,680 B/token KVGQA (standard; sliding_window 131072 exceeds max_position_embeddings = inactive)32,768 ctxmodels/qwen2-5-72b

verifiedhigh · 0.90

Qwen2.5-VL-7B-Instruct

dense 8.292B57,344 B/token KVGQA + mrope + windowed ViT128,000 ctxmodels/qwen2-5-vl-7b-instruct

verifiedhigh · 0.90

Qwen3 32B

dense 32B262,144 B/token KVGQA, uniform full attention on all 64 layers (no sliding window, no hybrid)40,960 ctxmodels/qwen3-32b

verifiedhigh · 0.85

Qwen3-235B-A22B

moe 22/235B active192,512 B/token KVGQA (standard)40,960 ctxmodels/qwen3-235b-a22b

verifiedhigh · 0.90

Qwen3-30B-A3B

moe 3.3/30.5B active98,304 B/token KVGQA (standard)40,960 ctxmodels/qwen3-30b-a3b

verifiedhigh · 0.90

Qwen3-8B

dense 8.19B147,456 B/token KVGQA (standard)40,960 ctxmodels/qwen3-8b

verifiedhigh · 0.90

Qwen3-Next 80B-A3B Instruct

hybrid-attention-ssm 80B24,576 B/token KVhybrid gated linear attention (Gated DeltaNet) in parallel with full GQA attention every 4th layer; 12 of 48 layers are full attention262,144 ctxmodels/qwen3-next-80b-a3b-instruct

verifiedhigh · 0.85

Qwen3-VL-8B-Instruct

dense 8.767B147,456 B/token KVGQA + mrope + DeepStack262,144 ctxmodels/qwen3-vl-8b-instruct

verifiedhigh · 0.90

Qwen3.5-122B-A10B

moe 10/121.349B active24,576 B/token KVGQA plus gated linear attention: 36 of 48 layers are linear_attention and 12 are full_attention, one every fourth layer262,144 ctxmodels/qwen3-5-122b-a10b

verifiedhigh · 0.86

Qwen3.5-35B-A3B

hybrid-attention-ssm 35B20,480 B/token KVhybrid: linear attention (Gated DeltaNet) with full attention every 4th layer (10 of 40)262,144 ctxmodels/qwen3-5-35b-a3b

verifiedhigh · 0.90

Qwen3.5-397B-A17B

moe 17/403.398B active30,720 B/token KVGQA plus gated linear attention: 45 of 60 layers are linear_attention and 15 are full_attention, one every fourth layer262,144 ctxmodels/qwen3-5-397b-a17b

verifiedhigh · 0.86

Qwen3.6-35B-A3B

moe 3/35.952B active20,480 B/token KVGQA plus gated linear attention: 30 of 40 layers are linear_attention and 10 are full_attention, one every fourth layer262,144 ctxmodels/qwen3-6-35b-a3b

verifiedhigh · 0.85

Qwen3.8-27B

hybrid-attention-ssm 27.781B65,536 B/token KVhybrid: Gated DeltaNet linear attention with full gated attention every 4th layer (16 of 64); partial RoPE262,144 ctxmodels/qwen3-8-27b

verifiedhigh · 0.86

Qwen3.8-Flash-Next

hybrid-attention-ssm 180B24,576 B/token KVhybrid: linear attention with full attention every 4th layer (12 of 48)262,144 ctxmodels/qwen3-8-flash-next

verifiedhigh · 0.90

RWKV-5 Eagle 7B (HF port)

recurrent 7.52B0 B/token KVnone - linear-attention recurrent (WKV) mixer4,096 ctxmodels/rwkv5-eagle-7b

verifiedmedium · 0.78

Seed-OSS-36B-Instruct

dense 36.6B262,144 B/token KVGQA (standard)524,288 ctxmodels/seed-oss-36b

verifiedhigh · 0.85

SmolLM2-1.7B-Instruct

dense 1.711B196,608 B/token KVMHA (Llama architecture)8,192 ctxmodels/smollm2-1-7b-instruct

verifiedhigh · 0.95

SmolLM3-3B

dense 3.0751B73,728 B/token KVGQA, NoPE on every 4th layer65,536 ctxmodels/smollm3-3b

verifiedhigh · 0.90

SmolVLM-Instruct (2.2B, Idefics3 arch)

dense 2.246B196,608 B/token KVMHA (no GQA)16,384 ctxmodels/smolvlm-2b-instruct

verifiedhigh · 0.90

SmolVLM2-2.2B-Instruct

dense 2.247B196,608 B/token KVMHA (no GQA)8,192 ctxmodels/smolvlm2-2-2b-instruct

verifiedhigh · 0.90

Snowflake Arctic Embed L

dense 0.334B0 B/token KVbidirectional encoder (BERT), full attention, no causal mask512 ctxmodels/snowflake-arctic-embed-l

verifiedhigh · 0.90

Solar Open 2

hybrid-attention-ssm 250.288B49,152 B/token KVhybrid: 36 linear-attention (KDA) layers + 12 softmax GQA layers at gqa_interval 3, with a GQA gate1,048,576 ctxmodels/solar-open2-250b

verifiedmedium · 0.65

Step 5 Preview

moe 27/600B active70,656 B/token KVGQA sliding-window hybrid with a sparse indexer on the full layers: 69 of 92 backbone layers are sliding_attention (window 512, 4 kv heads, head_dim 192) and 23 are full_attention, every fourth layer1,048,576 ctxmodels/step-5-preview

verifiedmedium · 0.78

Step-3.5-Flash

moe 11/196.8B active196,608 B/token KVGQA with interleaved sliding-window (36 of 48 layers, window 512) + full attention262,144 ctxmodels/step-3-5-flash

verifiedhigh · 0.85

TinyLlama-1.1B-Chat-v1.0

dense 1.1B22,528 B/token KVGQA (32 query / 4 KV, ratio 8)2,048 ctxmodels/tinyllama-1-1b-chat

verifiedhigh · 0.95

Unsloth DeepSeek-R1-Distill-Llama-8B (GGUF quant set)

dense 8.03026B131,072 B/token KVGQA131,072 ctxmodels/unsloth-deepseek-r1-distill-llama-8b-gguf

verifiedhigh · 0.90

Unsloth Hermes-4-405B (GGUF quant set)

dense 405.853B516,096 B/token KVGQA131,072 ctxmodels/unsloth-hermes-4-405b-gguf

verifiedhigh · 0.88

VeriLoop E2

hybrid-attention-ssm 27.781B65,536 B/token KVhybrid: Gated DeltaNet linear attention with full gated attention every 4th layer (16 of 64); partial RoPE262,144 ctxmodels/veriloop-e2

verifiedmedium · 0.78

Voxtral-Mini-3B-2507

dense 4.676B122,880 B/token KVGQA131,072 ctxmodels/voxtral-mini-3b-2507

verifiedhigh · 0.90

Whisper base

dense 0.074B12,288 B/token KVMHA, encoder-decoder (cross-attention)1,500 ctxmodels/whisper-base

verifiedhigh · 0.90

Whisper large-v2

dense 1.55B163,840 B/token KVMHA, encoder-decoder (cross-attention)1,500 ctxmodels/whisper-large-v2

verifiedhigh · 0.90

Whisper large-v3

dense 1.543B163,840 B/token KVMHA, encoder-decoder (cross-attention)1,500 ctxmodels/whisper-large-v3

verifiedhigh · 0.90

Whisper large-v3-turbo

dense 0.809B20,480 B/token KVMHA, encoder-decoder (cross-attention)1,500 ctxmodels/whisper-large-v3-turbo

verifiedhigh · 0.90

Whisper medium

dense 0.769B98,304 B/token KVMHA, encoder-decoder (cross-attention)1,500 ctxmodels/whisper-medium

verifiedhigh · 0.90

Whisper small

dense 0.244B36,864 B/token KVMHA, encoder-decoder (cross-attention)1,500 ctxmodels/whisper-small

verifiedhigh · 0.90

Whisper tiny

dense 0.039B6,144 B/token KVMHA, encoder-decoder (cross-attention)1,500 ctxmodels/whisper-tiny

verifiedhigh · 0.90

Xing4.0-29B-A4B

moe 4/31.215B active46,080 B/token KVMLA (kv_lora_rank 512, q_lora_rank 768, qk_nope_head_dim 128, qk_rope_head_dim 64, v_head_dim 128); expanded heads 32/32 are not the cached KV262,144 ctxMLA latent 512models/xing4-0-29b-a4b

verifiedmedium · 0.82

Most referenced

The records other records point at. Useful as an entry point: a heavily referenced accelerator or engine is one many decisions in this repo assume.

Record	Referenced by	Cites
Llama 3.3 70B	12	1
GPT-OSS 120B	9	9
Qwen3-235B-A22B	8	6
DeepSeek-V3	7	7
DeepSeek-V4-Flash	6	8
DeepSeek-R1	4	7
GLM-5.3	4	11
Kimi-K3	4	8