inference/research
verifiedhigh · 0.90updated 2026-10-05

Whisper tiny

dense 0.039B

Fields

id
models/whisper-tiny
Family
whisper
Vendor
OpenAI
Release year
2022
Architecture
dense
Params (B)
0.039
Active params (B)
0.039
Hidden size
384
Layers
4
KV heads
6
Head dim
64
GQA ratio
1
Max context
1,500
Context scaling method
native
Activation
gelu
Attention variant
MHA, encoder-decoder (cross-attention)
KV cache (bytes/token)
6,144
FLOPs per active token
78,000,000
Open weights
yes
License
apache-2.0
notes
ENCODER-DECODER. Smallest Whisper variant. ENCODER: 4 layers, d_model 384, 6 heads, encoder_ffn_dim 1536, gelu, num_mel_bins 80, max_source_positions 1500. DECODER: 4 layers, 6 heads, decoder_ffn_dim 1536, max_target_positions 448. KV CACHE: decoder self-attention 4 * 6 * 64 * 2 * 2 = 6,144 B/token = 6.0 KiB/token. Cross-attention over 1500 encoder frames: 4 * 6 * 64 * 1500 * 2 * 2 = 9,216,000 B = 8.8 MiB per in-flight request. The encoder is the compute-heavy part; the decoder is trivial. STREAMING: not natively supported (offline/batch model). KV CACHE CORRECTED (2026-10-05): kv_cache_bytes_per_token was 30,720 B/token and overstated the decoder self-attention cache by 5.00x. Correct value 6,144 B/token = 6.0 KiB/token, from 2 (K and V) x 4 decoder_layers x 6 decoder_attention_heads x 64 head_dim x 2 bytes (fp16). head_dim 64 is CORRECT here and was not changed: d_model 384 / decoder_attention_heads 6 = 64 for every Whisper variant, so 64 holds across the whole family and is not the source of the error. What was wrong was the multiplication itself - this record's own derivation line read '4 * 6 * 64 * 2 * 2 = 30,720', and that expression evaluates to 6,144, not 30,720. The formula was right and the result beside it was not, which is why the error survived: nothing in validate.py evaluates a notes expression against the field it claims to derive. Cross-attention over the fixed 1500-frame encoder span was already correct and is unchanged. See gotcha mdl-gotcha-derived-figures-field-disagrees-with-its-own-note.
task
speech-to-text

Notes

ENCODER-DECODER. Smallest Whisper variant. ENCODER: 4 layers, d_model 384, 6 heads, encoder_ffn_dim 1536, gelu, num_mel_bins 80, max_source_positions 1500. DECODER: 4 layers, 6 heads, decoder_ffn_dim 1536, max_target_positions 448. KV CACHE: decoder self-attention 4 * 6 * 64 * 2 * 2 = 6,144 B/token = 6.0 KiB/token. Cross-attention over 1500 encoder frames: 4 * 6 * 64 * 1500 * 2 * 2 = 9,216,000 B = 8.8 MiB per in-flight request. The encoder is the compute-heavy part; the decoder is trivial. STREAMING: not natively supported (offline/batch model). KV CACHE CORRECTED (2026-10-05): kv_cache_bytes_per_token was 30,720 B/token and overstated the decoder self-attention cache by 5.00x. Correct value 6,144 B/token = 6.0 KiB/token, from 2 (K and V) x 4 decoder_layers x 6 decoder_attention_heads x 64 head_dim x 2 bytes (fp16). head_dim 64 is CORRECT here and was not changed: d_model 384 / decoder_attention_heads 6 = 64 for every Whisper variant, so 64 holds across the whole family and is not the source of the error. What was wrong was the multiplication itself - this record's own derivation line read '4 * 6 * 64 * 2 * 2 = 30,720', and that expression evaluates to 6,144, not 30,720. The formula was right and the result beside it was not, which is why the error survived: nothing in validate.py evaluates a notes expression against the field it claims to derive. Cross-attention over the fixed 1500-frame encoder span was already correct and is unchanged. See gotcha mdl-gotcha-derived-figures-field-disagrees-with-its-own-note.

Cross-references

This record references no other records — it is a leaf. It is cited by 0.