Extending Context Window of Large Language Models via Positional Interpolation

Extending a transformer's context window by letting it see position indices beyond those it was trained on (extrapolation) produces catastrophically high attention scores that can destroy self-attention entirely.

Instead of extrapolating, INTERPOLATE: linearly down-scale the input position indices so the extended window maps onto the original trained range (positions p -> p / s for scale factor s). The paper's theoretical contribution is a bound showing the upper bound on the interpolated attention is at least ~600x smaller than that of extrapolation, which is why interpolation is stable and extrapolation is not. No architecture change - the same weights and the same kernels work, which is the point.

NOT A PERFORMANCE CLAIM. The one quantitative claim in the abstract is theoretical, not a speedup: the upper bound of interpolation is at least ~600x smaller than that of extrapolation. Quality result: models extended via Position Interpolation retain the original architecture and can reuse most pre-existing optimization and infrastructure.

arXiv ID 2306.15595 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. venue=arxiv-preprint: no venue claim in the arXiv comment (only 'Fix template issues'); OpenReview search for this exact title returns only a LATER paper by an overlapping team ('Exploring Context Window of LLMs via Decomposable Law', NeurIPS 2024) - a different paper, do not conflate. adoption=in-upstream-engine: position interpolation is the 'linear' rope_type, present in transformers ROPE_INIT_FUNCTIONS as 'linear' -> _compute_linear_scaling_rope_parameters, and in vLLM as LinearScalingRotaryEmbedding. Because the method is a pure position-index remap, ANY engine that accepts a scaling factor supports it without a new kernel - the strongest form of adoption and the reason this record is worth keeping. Note this is the paper that motivated YaRN and LongRoPE, both recorded separately here. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.