data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Every modality had its own bespoke self-supervised objective, so nothing transferred between speech, vision and text.

One objective for all modalities: a Transformer encoder is run on a masked view of the input, and an average of the top-K encoder layers is the target that the masked positions must predict via smooth L1 loss. Targets are contextualised latents carrying whole-sequence information rather than local units.

Reports new state-of-the-art or competitive results against modality-specific approaches on the major speech recognition, image classification and NLU benchmarks. No latency or throughput claim; quality baselines are per-modality benchmark leaderboards with no hardware specified.

Venue confirmed as ICML 2023: the PMLR v202 index carries this work under the retitled 'Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language', PMLR 202:1416-1429. The arXiv title (data2vec) and the venue title differ; the arXiv id is the correct handle. Compute-relevant detail absent from the abstract: the teacher is an average of top-K layers, so training needs two encoder passes per step, and at inference only the student encoder runs - a train-time-only cost, not a serving cost. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.