SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference

Existing pruning methods needed a learned pruner trained on extra data, adding parameters and fine-tuning cost.

TRAINING-FREE token optimisation: because visual tokens exist to support the text, it uses the model's own self-attention matrices - specifically the attention that TEXT tokens pay to visual tokens - to score visual-token importance, then prunes by rank with a per-layer adaptive sparsity ratio. Adds TOKEN RECYCLING, which compresses pruned tokens into more compact representations instead of discarding them.

On LLaVA with SparseVLM: 54% FLOP reduction and 37% decrease in CUDA LATENCY while retaining 97% of original accuracy. Both metrics and the retained-accuracy threshold are stated, which makes this the most complete cost claim among the vision-token papers. Baseline is plain LLaVA.

ICML 2025 (arXiv comment). The training-free property is what makes it attractive for serving - no pruner to host, no fine-tuning pipeline - and the latency measurement (not just FLOPs) is unusually honest for this literature. CAVEAT: a rank-based per-layer adaptive ratio means token count varies per input AND per layer, which forces either bucketing or padding in a batched engine; treat the 37% latency figure as single-request or small-batch. SparseVLM+ has since appeared in IEEE TPAMI, so the line is active. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.