High Fidelity Neural Audio Compression (EnCodec)

Neural codecs either matched speech-codec bitrates poorly or needed one model per bitrate, and training was slow and unstable.

Streaming encoder-decoder with a quantised latent space, trained end-to-end with a SINGLE multiscale-spectrogram adversary (cheaper and more stable than a stack of discriminators) plus a novel LOSS BALANCER in which each loss weight is expressed as the fraction of overall gradient it should own. Adds lightweight Transformer layers on top of the latent to squeeze a further 40%.

Superior to baseline codecs across all evaluated settings at both 24 kHz mono and 48 kHz stereo; lightweight Transformers compress the learned representation by up to a further 40% while staying faster than real time. Baselines are named (Opus, EVS) but no GPU is specified for the real-time claim.

arXiv v1 is explicitly marked 'Preprint' and crossref/openalex find no peer-reviewed venue - it should be cited as a technical report, not an ICASSP paper. This is the codec behind VALL-E and therefore the audio-token-rate reference point for generative speech work. Compared with SoundStream the architectural idea is similar (RVQ + streaming); EnCodec's differentiators are the single multi-scale adversary and the loss balancer, which are training-cost innovations rather than inference-cost ones - worth knowing, because the inference cost of the two is broadly comparable. VENUE CORRECTION + VENUE_TRACK 2026-10-04 (paper-fields pass, agent w3p): venue 'arxiv-preprint' -> 'iclr', venue_track 'main'. EVIDENCE: the STATIC proceedings index for this venue lists this exact title. Venue 'ICLR' paper page: ICLR 2024 poster 21753. The listing was corroborated by author overlap, not by title alone, because these titles are duplicated across venues and years and a title-only match is exactly the kind of thing that produced this repo's two previously-shipped wrong arXiv IDs. BOTH VALUES RETAINED PER THE REPO CONTRACT: the prior value was 'arxiv-preprint'. status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2210.13438). Method note: OpenReview's api2 hard-429s under load and pre-2024 venues are unreachable through it, so these were checked against static proceedings indexes (iclr.cc/virtual/<year>/papers.html and neurips.cc/virtual/<year>/papers.html, with the per-paper JSON-LD creditText field as the venue-year assertion). YEAR FIELD CORRECTED 2026-10-04 (same pass as the venue correction above): year 2022 -> 2024 to match the ICLR 2024 venue year. The prior value 2022 is the arXiv posting year and remains true of the preprint. status stays contested. This drift is why the venue correction and the year correction were done together rather than separately: setting venue without setting year leaves the record internally inconsistent. 2026-10-10: ICLR 2024 accepted titles index (idx3.py, final1.py: https://iclr.cc/virtual/2024/papers.html?filter=titles) confirms "High Fidelity Neural Audio Compression". arXiv API 2210.13438 (axc.py) comment states "Preprint". Source pap-mm-encodec notes cite arXiv preprint vs ICLR 2024 index. Sources disagree; status remains contested.