Careless Whisper: Speech-to-Text Hallucination Harms

Whisper silently emits fluent text that was never in the audio, which is a safety and data-integrity problem in any unattended deployment.

Audit rather than a method: measures the rate of hallucinated phrases in Whisper's output, themes them, and attributes them to language-model decoding behaviour (no VAD, non-speech segments, long internal non-speech durations). Establishes that the failure is a property of the decoder's unconditional text prior rather than of the acoustic encoder.

38% of the 1% of transcriptions containing entire hallucinated phrases include explicit harms; the hallucination rate is roughly 1% of all transcriptions. This is a correctness audit, not a speedup, and no hardware is involved.

VENUE CONFLICT, recorded deliberately: crossref places this title at ACM FAccT 2024 (DOI 10.1145/3630106.3658996) while the arXiv v2 comment says 'Accepted for IEEE ICASSP 2025'. Both are recorded here; the field is set to 'other' rather than guessing. Author list is truncated to the first six as returned by the arXiv metadata API and is not exhaustive. The practical inference consequence is direct: a VAD pre-filter that drops non-speech before the encoder is not a quality nicety, it is what suppresses the dominant failure mode, and it also cuts encoder prefill on silence. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. Peer-reviewed venue confirmed as ACM FAccT 2024 (same). This is an additive track annotation; `venue` is unchanged at 'other' and no status change is implied by the track alone. 2026-10-10: arXiv API 2402.08021 (axapi.py / axc.py: http://export.arxiv.org/api/query?id_list=2402.08021&max_results=1) confirms 5 authors [Allison Koenecke, Anna Seo Gyeong Choi, Katelyn X. Mei, Hilke Schellmann, Mona Sloane] and comment=NONE (the ICASSP comment is absent). Publisher deposit is ACM FAccT 2024 (DOI 10.1145/3630106.3658996, pages 1672-1681). Verified ACM FAccT 2024 publisher deposit (DOI 10.1145/3630106.3658996). Venue remains other due to schema enum gap (facct missing from schemas/paper.schema.json). Sources disagree (arXiv comment cited in notes vs FAccT deposit); status remains contested.