Whisper large-v3

ENCODER-DECODER. THE ENCODER IS THE COMPUTE-HEAVY PART AND THE DECODER IS TRIVIAL, and the config makes the split explicit: is_encoder_decoder true, max_source_positions 1500, max_target_positions 448. ENCODER (the cost): 32 layers, d_model 1280, 20 heads, encoder_ffn_dim 5120, gelu, run ONCE over the full 30s log-mel spectrogram (num_mel_bins 128, 3,000 frames at the Whisper hop) compressed to 1,500 encoder positions. Attention is bidirectional and there is NO KV CACHE for it -- it is a single full-sequence forward pass, not incremental decoding, so nothing about the encoder is cached between calls. Its ~0.64B parameters are the majority of the model and all of the attention work. DECODER (the trivial part): ALSO 32 layers here -- and this is the detail most write-ups get wrong. decoder_layers is 32 in this config, NOT 4. The famous shallow 4-layer decoder belongs to whisper-large-v3-TURBO (see that record), not to large-v3. decoder_attention_heads 20, decoder_ffn_dim 5120, max_target_positions 448. KV CACHE -- TWO SEPARATE RESIDENCES, and this field covers only the first: (a) DECODER SELF-ATTENTION KV, which grows with generated tokens: 32 layers * 20 kv_heads * 64 head_dim * 2 (K and V) * 2 bytes (fp16, per torch_dtype) = 163,840 B/token = 160 KiB/token. That is the value in kv_cache_bytes_per_token. (b) DECODER CROSS-ATTENTION K/V over the 1,500 encoder frames, which is CONSTANT per request and does not grow: 32 layers * 20 heads * 64 head_dim * 1500 frames * 2 * 2 bytes = 245,760,000 B = 234 MiB per in-flight request. This is the number that actually determines concurrency for Whisper, and it is larger than a full 448-token decode's self-attention KV (448 * 163,840 = 73 MiB) by 3x. DO NOT compare Whisper's memory footprint to an autoregressive text model using per-token KV alone: one request pins 234 MiB of cross-attention KV before a single token is emitted. FLOP PROFILE: encoder-decoder ASR has a fundamentally different shape from autoregressive decode. Precompute is a dense 32-layer bidirectional pass over 1,500 positions -- O(N^2) attention in the encoder, fully compute-bound and trivially parallel across the batch. Decode is at most 448 steps of a small-batch memory-bound GEMV over 160 KiB/token. For a 30s clip the encoder is >95% of the FLOPs. A GPU that is badly utilized decoding text will be near peak utilisation running the Whisper encoder. kv_cache_bytes_per_token is set from the DECODER ONLY because the encoder has no cache at all. flops_per_token_active is left NULL rather than 2*params: for an encoder-decoder that formula describes neither the encoder's 1,500-position precompute nor the decoder's cheap decode, and recording it would be actively misleading. STREAMING vs OFFLINE: this checkpoint is an OFFLINE model. Whisper's encoder is bidirectional over the full 30s window, so you cannot stream it -- you must buffer a full 30s chunk. Streaming ASR requires either a causal/chunked encoder or overlapping-window tricks with a latency penalty. params_b 1.543 = HF API safetensors total (1,543,490,560). SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=encoder-decoder. REQUIRED KERNEL CLASS: three distinct things in one graph, and this is the family where engines are least alike. (1) a bidirectional encoder over the full log-mel spectrogram with no KV cache at all - one full-sequence forward, O(N^2) attention, trivially batch-parallel, and >95% of the FLOPs for a 30s clip; (2) decoder SELF-attention KV, which does grow with generated tokens and is the value in kv_cache_bytes_per_token; (3) decoder CROSS-attention K/V over the encoder frames, which is CONSTANT per request at 234 MiB for large-v3 (32 layers x 20 heads x 64 head_dim x 1500 frames x 2 x 2 bytes) and does NOT grow - and that cross-attention block, not the per-token KV, is what determines Whisper concurrency, being 3x larger than a full 448-token decode's entire self-attention KV. An engine must therefore allocate two structurally different KV stores, and only one of them is sized by tokens. SUPPORTED BY (cited): vLLM has a native WhisperForConditionalGeneration row [[mdl2-vllm-supported-models-2026-10]], but encoder-decoder is explicitly NOT a first-class class there - 'For other model architectures not natively supported, in particular for Encoder-Decoder models, we recommend following a similar pattern by implementing support through the plugin system', and the only two plugin rows are BartForConditionalGeneration and Florence2ForConditionalGeneration [[mdl2-vllm-supported-models-2026-10]]. TensorRT-LLM has WhisperForConditionalGeneration plus T5ForConditionalGeneration, BartForConditionalGeneration and MBartForConditionalGeneration, and publishes the enc-dec constraints: 'Use the TRTLLM attention backend for encoder-decoder models; tensor parallelism also requires attention head counts divisible by the tensor parallel size. Chunked prefill is not supported for the encoder phase, so the complete encoder input must fit in the iteration token budget', and 'Whisper's feature-driven audio encoder runs eagerly' (encoder CUDA graphs cover only BART/mBART/T5) [[mdl2-trtllm-supported-models]]. vLLM's own ROCm caveat is that the encoder-decoder (enc-dec) FEATURE is unsupported on ROCm, and AMD GPUs are on the avoid_for list for enc-dec models [[vllm]]. SGLang serves openai/whisper-large-v3 on the audio/transcriptions route [[mdl2-sglang-multimodal-language-models]], and llama.cpp ships Whisper via whisper.cpp/Whisperfile rather than the main engine [[llamafile]] [[koboldcpp]] [[lemonade]]. HARD CONSTRAINTS CITED: chunked prefill does not apply to the encoder phase, so a long audio input must fit the whole iteration token budget [[mdl2-trtllm-supported-models]]; and enc-dec is one of the four features vLLM documents as unsupported on AMD alongside multi-step scheduling, async output and Marlin-kernel quantizations [[vllm]]. For the turbo checkpoint the decoder is 4 layers rather than 32, so its self-attention KV is 8x smaller while the 234 MiB cross-attention block is unchanged - the concurrency number barely moves.