SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

Cascaded speech systems (ASR -> text -> LLM -> TTS) cannot transfer knowledge between modalities, so speech understanding degrades relative to text.

Builds the SpeechInstruct cross-modal instruction dataset and trains with a three-stage strategy: modality-adaptation pre-training, then cross-modal instruction fine-tuning, then chain-of-modality instruction fine-tuning. Modality-specific input/output units are wrapped as discrete tokens and mixed into the LLM token stream.

arXiv preprint, self-described 'work in progress'; no peer-reviewed venue found. Do not treat the speech-LM cluster as production: SpeechGPT, SALMONN and Audio Flamingo are research artifacts whose checkpoints are demo-scale. The architectural lesson for inference is the token-rate issue again - representing speech as discrete units in the LLM token stream means audio length becomes LLM sequence length, so token rate is the dominant cost knob, and none of these papers report a hardware-level throughput comparison. VENUE CORRECTED 2026-10-04 (paper-verification pass): venue/year arxiv-preprint 2023 -> emnlp 2023. EVIDENCE: Title matches ACL Anthology EMNLP 2023 at score 0.984 as 2023.findings-emnlp.1055 (2295-title index; next best 0.62). The arXiv comment field reads only "work in progress", which is why it was recorded as a preprint — but the paper did appear in the EMNLP 2023 Findings proceedings. Note the venue nuance: this is a FINDINGS paper, not a main-track EMNLP paper. The repo has no way to express "findings" — venue "emnlp" plus this note is the closest faithful representation. Recording it as arxiv-preprint was an under-claim: it WAS peer reviewed. arxiv_id 2305.11000 is the preprint and is unchanged. Both values retained here deliberately; status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2305.11000). VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass): venue_track set to 'findings' from the ACL Anthology bib index (anthology.bib, 131650 entries) by normalized title match. Evidence: anthology id 2023.findings-emnlp.1055, booktitle 'Findings of the Association for Computational Linguistics: EMNLP 2023'. The Anthology id is the authoritative signal here and it is unambiguous, so this is an additive clarification of a track, not a change of venue - `venue` is untouched and no status change is implied. The reason this matters for inference readers: a Findings paper and a main-track paper at the same venue are not the same kind of evidence, and until this field existed the repo could not say which was which. 2026-10-10: Crossref deposit fetched via https://api.crossref.org/works/10.18653/v1/2023.findings-emnlp.1055 returned title 'SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities' matching container 'Findings of the Association for Computational Linguistics: EMNLP 2023'. Status resolved to verified with venue emnlp (conference).