Phi-4-multimodal-instruct

THREE-PART MODEL -- image, audio, and text all enter the same LLM. (1) IMAGE: CLIP ViT-L/14 at crop_size 448 (embd_layer.image_embd_layer.embedding_cls 'tune_image'), then the DYNAMIC CROPPING / HD TRANSFORM that decides prefill cost: use_hd_transform true, hd_transform_order 'sub_glb', preprocessor_config dynamic_hd 36, and image_token_compression_cls 'avg_pool_2d'. HD transform resizes the input to a large square and emits the global view plus up to 36 sub-crops of 336x336 (sub_glb = sub-crops then global). VISION TOKENS: each 336 crop through CLIP-14 is (336/14)^2 = 576 patches, then avg_pool_2d halves each spatial axis -> 144 tokens per crop. At dynamic_hd 36 that is up to 37 * 144 = 5,328 vision tokens per image, versus 144 for the single-crop default. DOMINANCE: VISION DOMINATES whenever HD transform engages (5,328 vs a ~100-token prompt, ~50:1); roughly balanced only on the default single 448 crop. The dynamic_hd knob is the single biggest cost lever in this record -- it moves vision tokens by 37x. The ViT is adapted by LoRA, not frozen: vision_lora r 256, lora_alpha 512, targeting self_attn.qkv_proj/o_proj and mlp.gate_up_proj/down_proj across all layers. (2) AUDIO: NeMo 'cascades' encoder, 24 blocks (num_blocks), d_model/attention_dim 1024, 16 heads, linear_units 1536, time_reduction 8, input_size 80 mel bins, T5 relative-attention bias with t5_bias_max_distance 500, causal true (left_chunk 18) -- so the audio encoder is chunk-aware and streamable, unlike Whisper's bidirectional one. audio_compression_rate 8. Also LoRA-adapted (speech_lora r 320). (3) LLM: Phi-4 dense decoder, 32 layers, hidden 3072, 24 heads / 8 KV heads, head_dim 128, intermediate 8192, vocab 200,064, partial_rotary_factor 0.75 (only 75% of head dims get RoPE), full_attn_mod 1, max_position 131,072 against original_max_position_embeddings 4096. ARCHITECTURE = dense (all three parts dense). CONTEXT_SCALING = LongRoPE: rope_scaling carries a long_factor array of per-dimension rescale factors, stretching 4,096 -> 131,072, with short_factor implied for the base window. rope_theta 1e4. KV CACHE: 32 * 8 * 128 * 2 * 2 = 131,072 B/token (128 KiB). At 131,072 ctx that is 16.0 GiB per sequence at bf16. params_b 5.574 = HF API safetensors total (5,574,460,384) -- the '4B' Phi-4 text model plus vision and audio encoders. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=multimodal. REQUIRED KERNEL CLASS: an encoder tower (ViT / SigLIP / ConvNet / audio encoder) running on a SEPARATE schedule from the decoder, plus a projector into the decoder's embedding space, plus - for the mrope and DeepStack families - a multi-axis rotary scheme in the decoder rather than plain 1-D RoPE. Three engine-side consequences, all cited. (1) The encoder tower is a separate compute path, so its throughput is a different problem from decode throughput and engines that disclaim encoder throughput are disclaiming this. vLLM says so in its own words: 'We currently support pooling models primarily for convenience. This is not guaranteed to provide any performance improvements over using Hugging Face Transformers or Sentence Transformers directly' [[vllm-pooling-models]] - that is the general statement this class inherits. TRT-LLM's answer is a documented mechanism: a Multimodal Encoder Side Stream that prefetches encoder work on a separate CUDA stream plus a per-model Multimodal Embeddings Cache, currently available only for seven architectures (Gemma4, Gemma4Unified, Mistral3, Qwen3VL, Qwen3VLMoe, Qwen3_5, Qwen3_5Moe) [[mdl2-trtllm-supported-models]]. (2) Several towers carry hard external dependencies: Gemma3nForConditionalGeneration 'depends on timm>=1.0.17 to make use of its MobileNet-v5 vision backbone' and 'is only supported on V1 due to shared KV caching' [[mdl2-vllm-supported-models-2026-10]]; MiniCPM-V 4.6 requires transformers>=5.7.0 because it was upstreamed as a native model type [[mdl2-trtllm-supported-models]]. (3) Hybrid-only multimodal models expose a memory knob unique to this class: vLLM's --language-model-only flag sets all supported multimodal modalities to 0 so the multimodal modules are not loaded, 'to free up more GPU memory for KV cache' for Llama-4, Step3, Mistral-3 and Qwen-3.5 [[mdl2-vllm-supported-models-2026-10]].SUPPORTED BY (cited): vLLM native rows include InternVLChatModel and InternVLForConditionalGeneration, LlavaOnevisionForConditionalGeneration, LlavaNextForConditionalGeneration, MolmoForCausalLM and Molmo2ForConditionalGeneration, PaliGemmaForConditionalGeneration, SmolVLMForConditionalGeneration, Glm4vForConditionalGeneration, Qwen2VL/Qwen3VL, MiniCPMV, Mllama, Phi4MM, Gemma4, NemotronH_Nano_VL_V2, VoxtralForConditionalGeneration (which requires mistral-common[audio]) and Qwen3OmniMoeThinkerForConditionalGeneration [[mdl2-vllm-supported-models-2026-10]]. SGLang's multimodal table covers InternLM2-VL, LLaVA-OneVision, LLaVA-NeXT(-Video), Ling-3.0-flash-VL, LFM2.5-VL, MiniCPM-V-2, GLM-4.5V, GLM-OCR, Qwen3-VL 235B-A22B and 30B-A3B, Qwen3-Omni-30B-A3B, Kimi-VL-A3B, MiMo-VL-7B-RL, Step3-VL-10B, dots.ocr, PaddleOCR-VL, Cosmos3, deepseek-vl2, Janus-Pro-7B, NVILA-8B and gemma-3-4b-it, plus openai/whisper-large-v3 and Qwen3-ASR-1.7B on the audio route [[mdl2-sglang-multimodal-language-models]]. TRT-LLM's multimodal matrix covers a narrower set with an explicit per-row modality string and per-row Untested cells [[mdl2-trtllm-supported-models]]. Older GGUF-side engines reach the class through llama.cpp's projector graph, which llama.cpp-embedding-server states it cannot serve on the embedding route [[llama-cpp-embedding-server]]. PER-MODALITY CAVEATS CITED: TRT-LLM marks Gemma4Unified and MiniCPM-V 4.6 Untested for CUDA Graph / chunked prefill, MiniMax-M3 Untested for attention DP and KV cache reuse, Qwen4Exp Untested across the board, and notes audio requires a checkpoint with a sound_config and is supported only on the full non-disaggregated path [[mdl2-trtllm-supported-models]]. Step-3.7's vision tower runs in BF16 even when the text decoder is quantized [[mdl2-trtllm-supported-models]] - a precision asymmetry that silently costs memory. vLLM's fallback path accepts image inputs only, with video 'to be added in a future release' [[mdl2-vllm-supported-models-2026-10]]. CONTEXT_SCALING_METHOD (other): SCHEME RECLASSIFIED (2026-10-04): was free text 'LongRoPE' had no enum value. rope_scaling type 'longrope'.