Whisper large-v3-turbo

ENCODER-DECODER, AND THIS IS THE MODEL WITH THE 4-LAYER DECODER. decoder_layers 4 vs 32 in large-v3 -- the whole point of turbo. ENCODER (unchanged, still the compute-heavy part): encoder_layers 32, d_model 1280, 20 heads, encoder_ffn_dim 5120, gelu, num_mel_bins 128, max_source_positions 1500. Identical to large-v3 and it DOMINATES the FLOPs: the 28 discarded decoder layers are ~0.73B of the 0.734B parameter reduction (1.543B -> 0.809B), and removing decoder layers removes decode work, NOT encoder work. DECODER: 4 layers, 20 heads, ffn 5120, max_target_positions 448. KV CACHE -- (a) DECODER SELF-ATTENTION, which is what kv_cache_bytes_per_token records: 4 layers * 20 kv_heads * 64 head_dim * 2 * 2 bytes (fp16) = 20,480 B/token = 20 KiB/token, exactly 1/8 of large-v3's 160 KiB because the layer count dropped 32 -> 4. (b) DECODER CROSS-ATTENTION over the 1,500 encoder frames, still CONSTANT per request: 4 * 20 * 64 * 1500 * 2 * 2 = 30,720,000 B = 29.3 MiB per in-flight request, exactly 1/8 of large-v3's 234 MiB. SO THE HEADLINE: turbo is ~2x faster end to end and uses ~8x less KV memory, but ONLY in the decode phase. The encoder -- >95% of the FLOPs for a typical 30s clip -- is bit-for-bit the same 32-layer pass. Anyone benchmarking turbo for throughput on short clips should expect a much smaller win than the 2x parameter reduction suggests, because the reduced part was never the bottleneck. The wins show up as lower time-to-first-token and as memory per concurrent request. FIELD-PAIR WARNING for anyone deriving this from num_layers: num_layers is 32 here because it is the ENCODER depth, and the ENCODER HAS NO KV CACHE. The 4-layer figure that produces kv_cache_bytes_per_token is decoder_layers, a different config key. So the repo-standard derivation num_layers * num_kv_heads * head_dim * 2 * 2 would give 163,840 and be wrong by 8x for this record. whisper-large-v3 has no such trap precisely because its decoder is also 32 layers. STREAMING: still OFFLINE. The bidirectional 1,500-position encoder is unchanged from large-v3, so turbo inherits the same 30s buffering requirement. Turbo did not make Whisper streamable. flops_per_token_active is NULL, not 2*params -- see the reasoning in the whisper-large-v3 record; for an encoder-decoder that formula describes neither phase. params_b 0.809 = HF API safetensors total (808,878,080). License differs from large-v3: MIT here vs Apache-2.0 there. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=encoder-decoder. REQUIRED KERNEL CLASS: three distinct things in one graph, and this is the family where engines are least alike. (1) a bidirectional encoder over the full log-mel spectrogram with no KV cache at all - one full-sequence forward, O(N^2) attention, trivially batch-parallel, and >95% of the FLOPs for a 30s clip; (2) decoder SELF-attention KV, which does grow with generated tokens and is the value in kv_cache_bytes_per_token; (3) decoder CROSS-attention K/V over the encoder frames, which is CONSTANT per request at 234 MiB for large-v3 (32 layers x 20 heads x 64 head_dim x 1500 frames x 2 x 2 bytes) and does NOT grow - and that cross-attention block, not the per-token KV, is what determines Whisper concurrency, being 3x larger than a full 448-token decode's entire self-attention KV. An engine must therefore allocate two structurally different KV stores, and only one of them is sized by tokens. SUPPORTED BY (cited): vLLM has a native WhisperForConditionalGeneration row [[mdl2-vllm-supported-models-2026-10]], but encoder-decoder is explicitly NOT a first-class class there - 'For other model architectures not natively supported, in particular for Encoder-Decoder models, we recommend following a similar pattern by implementing support through the plugin system', and the only two plugin rows are BartForConditionalGeneration and Florence2ForConditionalGeneration [[mdl2-vllm-supported-models-2026-10]]. TensorRT-LLM has WhisperForConditionalGeneration plus T5ForConditionalGeneration, BartForConditionalGeneration and MBartForConditionalGeneration, and publishes the enc-dec constraints: 'Use the TRTLLM attention backend for encoder-decoder models; tensor parallelism also requires attention head counts divisible by the tensor parallel size. Chunked prefill is not supported for the encoder phase, so the complete encoder input must fit in the iteration token budget', and 'Whisper's feature-driven audio encoder runs eagerly' (encoder CUDA graphs cover only BART/mBART/T5) [[mdl2-trtllm-supported-models]]. vLLM's own ROCm caveat is that the encoder-decoder (enc-dec) FEATURE is unsupported on ROCm, and AMD GPUs are on the avoid_for list for enc-dec models [[vllm]]. SGLang serves openai/whisper-large-v3 on the audio/transcriptions route [[mdl2-sglang-multimodal-language-models]], and llama.cpp ships Whisper via whisper.cpp/Whisperfile rather than the main engine [[llamafile]] [[koboldcpp]] [[lemonade]]. HARD CONSTRAINTS CITED: chunked prefill does not apply to the encoder phase, so a long audio input must fit the whole iteration token budget [[mdl2-trtllm-supported-models]]; and enc-dec is one of the four features vLLM documents as unsupported on AMD alongside multi-step scheduling, async output and Marlin-kernel quantizations [[vllm]]. For the turbo checkpoint the decoder is 4 layers rather than 32, so its self-attention KV is 8x smaller while the 234 MiB cross-attention block is unchanged - the concurrency number barely moves.