Kimi-Audio-7B-Instruct

CONTEXT SCALING: null is a VERIFIED value, checked against config rather than left blank. config.json declares rope_scaling null, rope_theta 1000000, sliding_window null, max_position_embeddings 8192. TWO-PART MODEL with the most unusual audio tokenization in this slice, and the only one that uses MiMo. (1) AUDIO (the compute-heavy part): use_whisper_feature true, so it consumes Whisper-style acoustic features, but the audio pathway is NOT a plain Whisper encoder bolted onto an LLM. The distinctive fields are the MiMo ones: kimia_mimo_layers 6, kimia_mimo_transformer_from_layer_index 21, kimia_mimo_audiodelaytokens 6, kimia_adaptor_input_dim 5120, num_audio_special_tokens 512, kimia_audio_output_vocab 16896, kimia_token_offset 152064, kimia_media_begin/end 151661/151663. Reading these together: audio is routed through a 6-layer MiMo audio model that is injected starting at LLM layer 21 (not at the input), and the model can emit its own 16,896-token AUDIO vocabulary in addition to the 152,064-token text vocabulary, living in a DISJOINT token space via kimia_token_offset 152064. That is why total vocab is 168,448 = 152,064 + 16,896 - 512. So unlike every other audio record here, Kimi-Audio is not a pure ASR model: it can reason over audio and generate audio tokens, i.e. a speech-in / speech-out model built on a text LLM. AUDIO POSITIONS are governed by num_audio_special_tokens 512 rather than a fixed 1,500-frame Whisper output, which is a much more compact audio representation than Whisper's. (2) LLM: dense decoder, 28 layers, hidden 3584, 28 heads / 4 KV heads, head_dim 128, intermediate 18944, max_position 8192, rope_theta 1e6, rope_scaling null, tie_word_embeddings false, rms_norm_eps 1e-6. KV geometry independently confirmed from tensor shapes: model.layers.0.self_attn.k_proj.weight is [512, 3584] = 4 KV heads x 128, matching the declared num_key_value_heads 4; q_proj is [3584,3584]. lm_head [168448,3584]. ARCHITECTURE = dense. KV CACHE: 28 * 4 * 128 * 2 * 2 = 57,344 B/token (56 KiB). The best KV-per-token of any audio-capable model in this slice -- 9.1x smaller than Qwen2-Audio's 512 KiB, for a larger model. Qwen2-Audio's MHA choice and Kimi-Audio's 7:1 GQA are the whole spread. flops_per_token_active = 2 * 9.766e9 = 1.95e10 is set here because this model decodes TEXT tokens through a normal autoregressive LLM, so the formula is meaningful; note it does NOT include the audio pathway, which is a separate precompute pass. STREAMING: audio enters as a fixed block of special tokens with no causal-chunked encoder declared, so treat as OFFLINE. params_b 9.766 = HF API safetensors total (9,766,336,640) -- the '7B' name counts the text backbone; the MiMo audio model plus the enlarged 168,448-token vocabulary account for the extra ~2.8B. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=multimodal. REQUIRED KERNEL CLASS: an encoder tower (ViT / SigLIP / ConvNet / audio encoder) running on a SEPARATE schedule from the decoder, plus a projector into the decoder's embedding space, plus - for the mrope and DeepStack families - a multi-axis rotary scheme in the decoder rather than plain 1-D RoPE. Three engine-side consequences, all cited. (1) The encoder tower is a separate compute path, so its throughput is a different problem from decode throughput and engines that disclaim encoder throughput are disclaiming this. vLLM says so in its own words: 'We currently support pooling models primarily for convenience. This is not guaranteed to provide any performance improvements over using Hugging Face Transformers or Sentence Transformers directly' [[vllm-pooling-models]] - that is the general statement this class inherits. TRT-LLM's answer is a documented mechanism: a Multimodal Encoder Side Stream that prefetches encoder work on a separate CUDA stream plus a per-model Multimodal Embeddings Cache, currently available only for seven architectures (Gemma4, Gemma4Unified, Mistral3, Qwen3VL, Qwen3VLMoe, Qwen3_5, Qwen3_5Moe) [[mdl2-trtllm-supported-models]]. (2) Several towers carry hard external dependencies: Gemma3nForConditionalGeneration 'depends on timm>=1.0.17 to make use of its MobileNet-v5 vision backbone' and 'is only supported on V1 due to shared KV caching' [[mdl2-vllm-supported-models-2026-10]]; MiniCPM-V 4.6 requires transformers>=5.7.0 because it was upstreamed as a native model type [[mdl2-trtllm-supported-models]]. (3) Hybrid-only multimodal models expose a memory knob unique to this class: vLLM's --language-model-only flag sets all supported multimodal modalities to 0 so the multimodal modules are not loaded, 'to free up more GPU memory for KV cache' for Llama-4, Step3, Mistral-3 and Qwen-3.5 [[mdl2-vllm-supported-models-2026-10]].SUPPORTED BY (cited): vLLM native rows include InternVLChatModel and InternVLForConditionalGeneration, LlavaOnevisionForConditionalGeneration, LlavaNextForConditionalGeneration, MolmoForCausalLM and Molmo2ForConditionalGeneration, PaliGemmaForConditionalGeneration, SmolVLMForConditionalGeneration, Glm4vForConditionalGeneration, Qwen2VL/Qwen3VL, MiniCPMV, Mllama, Phi4MM, Gemma4, NemotronH_Nano_VL_V2, VoxtralForConditionalGeneration (which requires mistral-common[audio]) and Qwen3OmniMoeThinkerForConditionalGeneration [[mdl2-vllm-supported-models-2026-10]]. SGLang's multimodal table covers InternLM2-VL, LLaVA-OneVision, LLaVA-NeXT(-Video), Ling-3.0-flash-VL, LFM2.5-VL, MiniCPM-V-2, GLM-4.5V, GLM-OCR, Qwen3-VL 235B-A22B and 30B-A3B, Qwen3-Omni-30B-A3B, Kimi-VL-A3B, MiMo-VL-7B-RL, Step3-VL-10B, dots.ocr, PaddleOCR-VL, Cosmos3, deepseek-vl2, Janus-Pro-7B, NVILA-8B and gemma-3-4b-it, plus openai/whisper-large-v3 and Qwen3-ASR-1.7B on the audio route [[mdl2-sglang-multimodal-language-models]]. TRT-LLM's multimodal matrix covers a narrower set with an explicit per-row modality string and per-row Untested cells [[mdl2-trtllm-supported-models]]. Older GGUF-side engines reach the class through llama.cpp's projector graph, which llama.cpp-embedding-server states it cannot serve on the embedding route [[llama-cpp-embedding-server]]. PER-MODALITY CAVEATS CITED: TRT-LLM marks Gemma4Unified and MiniCPM-V 4.6 Untested for CUDA Graph / chunked prefill, MiniMax-M3 Untested for attention DP and KV cache reuse, Qwen4Exp Untested across the board, and notes audio requires a checkpoint with a sound_config and is supported only on the full non-disaggregated path [[mdl2-trtllm-supported-models]]. Step-3.7's vision tower runs in BF16 even when the text decoder is quantized [[mdl2-trtllm-supported-models]] - a precision asymmetry that silently costs memory. vLLM's fallback path accepts image inputs only, with video 'to be added in a future release' [[mdl2-vllm-supported-models-2026-10]]. CONTEXT_SCALING_METHOD (native): rope_scaling null at theta 1e6 over 8192.