Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models

Open-weight VLMs relied on synthetic data distilled from proprietary VLMs, so the community could not build or audit them from scratch.

Introduces PixMo datasets (detailed captions, free-form QA, and a 2D POINTING dataset) collected without any external VLM. The model's distinctive architecture choice is a DELIBERATELY SMALL, VARIABLE-SIZE POOL OF VISION TOKENS: the ViT encodes an image-caption pair in one pass and the array-pooling layer pools spatial image tokens into a small set of multimodal embeddings using the paired text as a guide, rather than emitting a full patch grid to the LM.

Venue CVPR 2025 (crossref container '2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition'). Reported as the strongest open-weights/open-data VLM at the time and reported to beat Claude 3.5 Sonnet and Gemini 1.5 Pro/Flash on its evaluations, with only GPT-4o ahead - all vendor self-reported human and benchmark evaluation, so treat with the usual scepticism. THE GENUINE EFFICIENCY INNOVATION, as the task brief anticipated: 2D pooling driven by paired text yields a dramatically smaller LM-side token count than a patch-grid projector. Cost moves from the LM prefill (which is what vision tokens actually dominate) back into the vision encoder, and that is the right direction for a serving system because the encoder can be cached per image while LM prefill cannot be. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: cvpr has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.