SmolVLM-Instruct (2.2B, Idefics3 arch)

TWO-PART MODEL, the Idefics3 architecture, and the EFFICIENCY DESIGN is the record here. (1) Vision: SigLIP tower, 27 layers, hidden 1152, 16 heads, patch 14, image_size 384, gelu_pytorch_tanh. Naive cost would be (384/14)^2 = 729 patches. Instead: pixel_shuffle / scale_factor 3 compresses before the PERCEIVER RESAMPLER (perceiver_config nested in text_config: resampler_n_latents 64, resampler_depth 6, resampler_head_dim 96, resampler_n_heads 16, num_key_value_heads 1). VISION TOKENS = FIXED, AT 64 PER IMAGE, REGARDLESS OF RESOLUTION. image_seq_len is 81 (64 image tokens + 16 text tokens carrying the global view). vision_config.size.longest_edge 1920 vs max_image_size.longest_edge 384 means the image is tiled for the encoder and then collapsed to those 64 latents, so the LLM never sees more than 64 vision tokens however large the input. DOMINANCE: TEXT DOMINATES, decisively and by design -- 64 vision tokens is smaller than a single sentence. This is the deliberate opposite of Gemma 3 (4,096 tokens) and Pixtral (4,096 per tile), and it is why SmolVLM is the cheapest VLM in this slice to prefill. The tradeoff is lost detail. (2) LLM: VLlama3 dense decoder, 24 layers, hidden 2048, 32 heads AND 32 KV heads, head_dim 64, intermediate 8192, vocab 49,155 (Llama-3 tokenizer), max_position 16,384. ARCHITECTURE = dense. KV CACHE: 24 * 32 * 64 * 2 * 2 = 196,608 B/token (192 KiB) -- MHA, no GQA at all. This is 3.4x Qwen2.5-VL-7B's per-token KV on a model a third the size, and it is the one genuinely bad serving number in this record. At 16,384 ctx that is 3.0 GiB per sequence. If you shrink vision tokens to 64, spend the budget on KV head reduction instead. params_b 2.246 = HF API safetensors total (2,246,272,880). SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=multimodal. REQUIRED KERNEL CLASS: an encoder tower (ViT / SigLIP / ConvNet / audio encoder) running on a SEPARATE schedule from the decoder, plus a projector into the decoder's embedding space, plus - for the mrope and DeepStack families - a multi-axis rotary scheme in the decoder rather than plain 1-D RoPE. Three engine-side consequences, all cited. (1) The encoder tower is a separate compute path, so its throughput is a different problem from decode throughput and engines that disclaim encoder throughput are disclaiming this. vLLM says so in its own words: 'We currently support pooling models primarily for convenience. This is not guaranteed to provide any performance improvements over using Hugging Face Transformers or Sentence Transformers directly' [[vllm-pooling-models]] - that is the general statement this class inherits. TRT-LLM's answer is a documented mechanism: a Multimodal Encoder Side Stream that prefetches encoder work on a separate CUDA stream plus a per-model Multimodal Embeddings Cache, currently available only for seven architectures (Gemma4, Gemma4Unified, Mistral3, Qwen3VL, Qwen3VLMoe, Qwen3_5, Qwen3_5Moe) [[mdl2-trtllm-supported-models]]. (2) Several towers carry hard external dependencies: Gemma3nForConditionalGeneration 'depends on timm>=1.0.17 to make use of its MobileNet-v5 vision backbone' and 'is only supported on V1 due to shared KV caching' [[mdl2-vllm-supported-models-2026-10]]; MiniCPM-V 4.6 requires transformers>=5.7.0 because it was upstreamed as a native model type [[mdl2-trtllm-supported-models]]. (3) Hybrid-only multimodal models expose a memory knob unique to this class: vLLM's --language-model-only flag sets all supported multimodal modalities to 0 so the multimodal modules are not loaded, 'to free up more GPU memory for KV cache' for Llama-4, Step3, Mistral-3 and Qwen-3.5 [[mdl2-vllm-supported-models-2026-10]].SUPPORTED BY (cited): vLLM native rows include InternVLChatModel and InternVLForConditionalGeneration, LlavaOnevisionForConditionalGeneration, LlavaNextForConditionalGeneration, MolmoForCausalLM and Molmo2ForConditionalGeneration, PaliGemmaForConditionalGeneration, SmolVLMForConditionalGeneration, Glm4vForConditionalGeneration, Qwen2VL/Qwen3VL, MiniCPMV, Mllama, Phi4MM, Gemma4, NemotronH_Nano_VL_V2, VoxtralForConditionalGeneration (which requires mistral-common[audio]) and Qwen3OmniMoeThinkerForConditionalGeneration [[mdl2-vllm-supported-models-2026-10]]. SGLang's multimodal table covers InternLM2-VL, LLaVA-OneVision, LLaVA-NeXT(-Video), Ling-3.0-flash-VL, LFM2.5-VL, MiniCPM-V-2, GLM-4.5V, GLM-OCR, Qwen3-VL 235B-A22B and 30B-A3B, Qwen3-Omni-30B-A3B, Kimi-VL-A3B, MiMo-VL-7B-RL, Step3-VL-10B, dots.ocr, PaddleOCR-VL, Cosmos3, deepseek-vl2, Janus-Pro-7B, NVILA-8B and gemma-3-4b-it, plus openai/whisper-large-v3 and Qwen3-ASR-1.7B on the audio route [[mdl2-sglang-multimodal-language-models]]. TRT-LLM's multimodal matrix covers a narrower set with an explicit per-row modality string and per-row Untested cells [[mdl2-trtllm-supported-models]]. Older GGUF-side engines reach the class through llama.cpp's projector graph, which llama.cpp-embedding-server states it cannot serve on the embedding route [[llama-cpp-embedding-server]]. PER-MODALITY CAVEATS CITED: TRT-LLM marks Gemma4Unified and MiniCPM-V 4.6 Untested for CUDA Graph / chunked prefill, MiniMax-M3 Untested for attention DP and KV cache reuse, Qwen4Exp Untested across the board, and notes audio requires a checkpoint with a sound_config and is supported only on the full non-disaggregated path [[mdl2-trtllm-supported-models]]. Step-3.7's vision tower runs in BF16 even when the text decoder is quantized [[mdl2-trtllm-supported-models]] - a precision asymmetry that silently costs memory. vLLM's fallback path accepts image inputs only, with video 'to be added in a future release' [[mdl2-vllm-supported-models-2026-10]]. CONTEXT_SCALING_METHOD (native): rope_scaling null at theta 273768 over 16384.