Learning Transferable Visual Models From Natural Language Supervision (CLIP)

Vision encoders trained on a fixed label set cannot represent any concept outside that set without new labelled data.

Dual-tower contrastive training on 400M internet (image, text) pairs: each tower is a modified ResNet or ViT producing a vector, and a learned temperature-scaled cosine similarity is trained symmetrically with InfoNCE. The decoder is DISCARDED after pre-training, so inference is two independent encoder passes plus a dot product - no autoregressive cross-attention, hence no KV cache for the vision side.

Venue ICML 2021, confirmed at proceedings.mlr.press v139/radford21a. INFERENCE RELEVANCE: CLIP is still the default image tower in production VLMs and CLIP-based retrieval systems, and the projector-free architecture is why VLM serving stacks run the vision encoder once per image with no cache. What CLIP costs is the resolution/patch grid: a 224px input is 7x7=49 patches for a ViT-B/32, 196 for ViT-L/14 at 336px, and token count scales with area, which is the root of the vision-token-cost problem that VisionZip/SparseVLM/DynamicViT attack. The paper reports no throughput or latency numbers; the 400M-pair figure is a training-data scale, not a speed claim. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.