Visual Instruction Tuning (LLaVA)

LLMs had strong zero-shot behaviour from instruction tuning but vision-language models had not been shown to benefit from the same recipe.

Uses a language-only GPT-4 to generate multimodal instruction-following data, then instruction-tunes an end-to-end LMM connecting a vision encoder to an LLM. The architectural contribution that matters here is deliberately minimal: the projector is a single linear layer mapping CLIP image features into the LM embedding space, so no resampler and no learned query tokens.

Yields an 85.1% relative score versus GPT-4 on a synthetic multimodal instruction-following dataset; LLaVA + GPT-4 synergy reaches 92.53% on ScienceQA. Quality-only metrics, no latency or throughput disclosure.

NeurIPS 2023 Oral (arXiv comment). LLaVA is the most-influenced open VLM architecture in production - its weights are the base of a large share of deployed VLMs and it is the reference implementation for vLLM/SGLang vision paths. INFERENCE NOTE: because the projector is linear and there is no resampler, LLaVA passes the vision encoder's FULL patch sequence to the LM, so its prefill cost is the highest in this family and scales with image resolution. That is exactly the gap LLaVA-NeXT's anyres tiling, and then VisionZip/SparseVLM pruning, close. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.