LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

Open LMMs were built for single images, so multi-image, video and 3D inputs could not be handled by one model.

Treats interleaved multimodal data as a general template covering multi-image, multi-frame, multi-view and multi-patch cases, trained on the 1,177.6k-sample M4-Instruct dataset. Builds on the LLaVA-NeXT line, whose any-resolution training and inference-time tiling are what handle multi-patch inputs.

CAUTION ON IDENTITY: the paper commonly cited as 'LLaVA-NeXT: A Stronger Instruction Tuning LMM with Extended Data and Recipe' is NOT separately indexed on arXiv under that title - a title search returns nothing. This record therefore points at the Interleave variant, which IS indexed, and the source note says so. Any citation of 'LLaVA-NeXT' should be checked against the specific arXiv id before being trusted. THE COST MECHANISM: any-resolution processing splits an image into a global view plus multiple local crops, and each crop becomes its own set of visual tokens, so token count MULTIPLIES with the number of tiles. Anyres tiling raises resolution and therefore raises prefill cost superlinearly - this is the technique that vision-token pruning was invented to make affordable. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.