SALMONN: Towards Generic Hearing Abilities for Large Language Models

LLMs are text-only; cascaded ASR pipelines lose paralinguistic information and cannot handle music or audio events.

Whisper-based encoder plus a Whisper/HuBERT audio feature stack projected into the LLM embedding space, with Q-Former-style Perceiver resampler blocks compressing variable-length audio into a small fixed number of vectors before the LLM, plus a lightweight 'tuning' stage to unlock cross-modal emergent ability.

CVPR 2024 (confirmed via crossref). CAUTION on citation: arXiv admin flagged video-SALMONN (2406.15704) for substantial text overlap with 2310.05863, so treat 2310.13289 and the video-SALMONN line as one research cluster rather than independent corroboration. The resampler is the inference-relevant mechanism: compressing audio to a fixed token count decouples LLM prefill cost from audio duration, which is the same lever the VLM projectors pull on the vision side. VENUE CORRECTION + VENUE_TRACK 2026-10-04 (paper-fields pass, agent w3p): venue 'cvpr' -> 'iclr', venue_track 'main'. EVIDENCE: the STATIC proceedings index for this venue lists this exact title. Venue 'ICLR' paper page: ICLR 2024 poster 19600. The listing was corroborated by author overlap, not by title alone, because these titles are duplicated across venues and years and a title-only match is exactly the kind of thing that produced this repo's two previously-shipped wrong arXiv IDs. BOTH VALUES RETAINED PER THE REPO CONTRACT: the prior value was 'cvpr'. status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2310.13289). Method note: OpenReview's api2 hard-429s under load and pre-2024 venues are unreachable through it, so these were checked against static proceedings indexes (iclr.cc/virtual/<year>/papers.html and neurips.cc/virtual/<year>/papers.html, with the per-paper JSON-LD creditText field as the venue-year assertion). YEAR FIELD CORRECTED 2026-10-04 (same pass as the venue correction above): year 2024 -> 2024 to match the ICLR 2024 venue year. The prior value 2024 is the arXiv posting year and remains true of the preprint. status stays contested. This drift is why the venue correction and the year correction were done together rather than separately: setting venue without setting year leaves the record internally inconsistent. 2026-10-10: ICLR 2024 virtual poster (final1.py, idx5.py: https://iclr.cc/virtual/2024/poster/19600) and titles listing (idx4.py, idx6.py: https://iclr.cc/virtual/2024/papers.html?filter=titles) confirm "SALMONN: Towards Generic Hearing Abilities for Large Language Models". arXiv API 2310.13289 (axc.py) verified. Source pap-mm-salmonn notes cite CVPR 2024 vs ICLR 2024 poster 19600. Sources disagree; status remains contested.