DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification

Vision transformers compute attention over every patch even though the final prediction depends on only a subset of informative tokens.

A lightweight prediction module scores each token's importance from current features and hierarchically prunes low-scoring tokens at several depths. Trained end-to-end via an attention MASKING strategy that blocks a token's interactions, which makes pruning differentiable. Crucially, pruning is UNSTRUCTURED, so the surviving tokens stay contiguous in memory and remain hardware friendly.

Hierarchically pruning 66% of input tokens reduces FLOPs by 31%-37% and improves throughput by over 40% with an accuracy drop within 0.5% across several vision transformers. This is the rare vision paper with BOTH a FLOP reduction and a measured THROUGHPUT gain - and the accuracy threshold is stated.

NeurIPS 2021 (arXiv comment; the NeurIPS 2021 proceedings index contains the title). Recorded as the origin of DATA-DEPENDENT token count: cost varies per input rather than per model, which is a scheduling problem for a serving system (batched requests now need padding to the batch's largest surviving token set). Its direct VLM successors - pap records visionzip and sparsevlm - keep the data-dependent idea and move the pruning decision to the vision-LM boundary where it actually saves LM prefill. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.