Qwen2-Audio-7B-Instruct

CONTEXT SCALING: null is a VERIFIED value, checked against config rather than left blank. config.json declares rope_scaling null, rope_theta 10000, sliding_window 32768, max_position_embeddings 8192. TWO-PART MODEL, audio-in / text-out, general audio understanding rather than pure ASR (it handles voice chat and open-ended audio analysis, not just transcription). (1) AUDIO ENCODER (the compute-heavy part): audio_config is a WHISPER-LARGE ENCODER verbatim -- d_model 1280, encoder_layers 32, encoder_attention_heads 20, encoder_ffn_dim 5120, gelu, num_mel_bins 128, max_source_positions 1500. Qwen reused Whisper's audio front-end rather than training a new one. Confirmed independently from tensor shapes: audio_tower.layers.0.self_attn.q_proj.weight is [1280,1280] and layers.31 likewise, so the encoder really is 32 layers at d_model 1280. AUDIO TOKENS: 1,500 encoder positions per 30s window. NO KV CACHE for the encoder -- single bidirectional pass. (2) LLM: Qwen2-7B dense decoder. IMPORTANT: text_config in this repo OMITS num_attention_heads and num_key_value_heads, so these were read from safetensors tensor shapes in the repo: language_model.model.layers.0.self_attn.q_proj.weight is [4096,4096] and k_proj.weight is also [4096,4096] at layers 0, 7 AND 31 -- K is NOT narrowed, so this is plain MHA with 32 heads of head_dim 128, no GQA and no MQA. lm_head [156032,4096] confirms hidden 4096 and vocab 156,032; layers.31.mlp.gate_proj [11008,4096] confirms intermediate 11,008 and 32 layers. Method note: this tensor-shape method was validated against Qwen2.5-VL-7B (k_proj [512,3584] = exactly 4 KV heads x 128) and Kimi-Audio (k_proj [512,3584], matching its declared 4 KV heads) before being used to conclude MHA here. activation is NULL because text_config states no hidden_act. ARCHITECTURE = dense. use_mrope false. KV CACHE: 32 * 32 * 128 * 2 * 2 = 524,288 B/token = 512 KiB/token. THIS IS THE LARGEST KV-PER-TOKEN OF ANY RECORD IN THIS SLICE, by 33% over the next worst (Phi-3.5-vision and Gemma 3 12B at 393,216 B). Full MHA on a 7B model is an unusually costly choice in 2024-25 and it is a direct consequence of the text_config omitting the GQA fields. At 8,192 ctx a single request pins 4.0 GiB of KV, and because the 1,500 audio positions are an LLM prefix, a 30s audio prompt already commits 1,500 * 524,288 = 750 MiB before any text is generated. STREAMING: OFFLINE -- bidirectional Whisper encoder, fixed 30s window. params_b 8.397 = HF API safetensors total (8,397,094,912). SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=multimodal. REQUIRED KERNEL CLASS: an encoder tower (ViT / SigLIP / ConvNet / audio encoder) running on a SEPARATE schedule from the decoder, plus a projector into the decoder's embedding space, plus - for the mrope and DeepStack families - a multi-axis rotary scheme in the decoder rather than plain 1-D RoPE. Three engine-side consequences, all cited. (1) The encoder tower is a separate compute path, so its throughput is a different problem from decode throughput and engines that disclaim encoder throughput are disclaiming this. vLLM says so in its own words: 'We currently support pooling models primarily for convenience. This is not guaranteed to provide any performance improvements over using Hugging Face Transformers or Sentence Transformers directly' [[vllm-pooling-models]] - that is the general statement this class inherits. TRT-LLM's answer is a documented mechanism: a Multimodal Encoder Side Stream that prefetches encoder work on a separate CUDA stream plus a per-model Multimodal Embeddings Cache, currently available only for seven architectures (Gemma4, Gemma4Unified, Mistral3, Qwen3VL, Qwen3VLMoe, Qwen3_5, Qwen3_5Moe) [[mdl2-trtllm-supported-models]]. (2) Several towers carry hard external dependencies: Gemma3nForConditionalGeneration 'depends on timm>=1.0.17 to make use of its MobileNet-v5 vision backbone' and 'is only supported on V1 due to shared KV caching' [[mdl2-vllm-supported-models-2026-10]]; MiniCPM-V 4.6 requires transformers>=5.7.0 because it was upstreamed as a native model type [[mdl2-trtllm-supported-models]]. (3) Hybrid-only multimodal models expose a memory knob unique to this class: vLLM's --language-model-only flag sets all supported multimodal modalities to 0 so the multimodal modules are not loaded, 'to free up more GPU memory for KV cache' for Llama-4, Step3, Mistral-3 and Qwen-3.5 [[mdl2-vllm-supported-models-2026-10]].SUPPORTED BY (cited): vLLM native rows include InternVLChatModel and InternVLForConditionalGeneration, LlavaOnevisionForConditionalGeneration, LlavaNextForConditionalGeneration, MolmoForCausalLM and Molmo2ForConditionalGeneration, PaliGemmaForConditionalGeneration, SmolVLMForConditionalGeneration, Glm4vForConditionalGeneration, Qwen2VL/Qwen3VL, MiniCPMV, Mllama, Phi4MM, Gemma4, NemotronH_Nano_VL_V2, VoxtralForConditionalGeneration (which requires mistral-common[audio]) and Qwen3OmniMoeThinkerForConditionalGeneration [[mdl2-vllm-supported-models-2026-10]]. SGLang's multimodal table covers InternLM2-VL, LLaVA-OneVision, LLaVA-NeXT(-Video), Ling-3.0-flash-VL, LFM2.5-VL, MiniCPM-V-2, GLM-4.5V, GLM-OCR, Qwen3-VL 235B-A22B and 30B-A3B, Qwen3-Omni-30B-A3B, Kimi-VL-A3B, MiMo-VL-7B-RL, Step3-VL-10B, dots.ocr, PaddleOCR-VL, Cosmos3, deepseek-vl2, Janus-Pro-7B, NVILA-8B and gemma-3-4b-it, plus openai/whisper-large-v3 and Qwen3-ASR-1.7B on the audio route [[mdl2-sglang-multimodal-language-models]]. TRT-LLM's multimodal matrix covers a narrower set with an explicit per-row modality string and per-row Untested cells [[mdl2-trtllm-supported-models]]. Older GGUF-side engines reach the class through llama.cpp's projector graph, which llama.cpp-embedding-server states it cannot serve on the embedding route [[llama-cpp-embedding-server]]. PER-MODALITY CAVEATS CITED: TRT-LLM marks Gemma4Unified and MiniCPM-V 4.6 Untested for CUDA Graph / chunked prefill, MiniMax-M3 Untested for attention DP and KV cache reuse, Qwen4Exp Untested across the board, and notes audio requires a checkpoint with a sound_config and is supported only on the full non-disaggregated path [[mdl2-trtllm-supported-models]]. Step-3.7's vision tower runs in BF16 even when the text decoder is quantized [[mdl2-trtllm-supported-models]] - a precision asymmetry that silently costs memory. vLLM's fallback path accepts image inputs only, with video 'to be added in a future release' [[mdl2-vllm-supported-models-2026-10]].