LongRoPE2: Near-Lossless LLM Context Window Scaling

Existing RoPE rescaling suffers persistent out-of-distribution behaviour at long context, and the paper's diagnosis is that the higher RoPE dimensions are UNDER-TRAINED because most training tokens never exercise them.

Three parts: (1) the insufficient-training hypothesis for high RoPE dimensions as the root cause of OOD failure; (2) a RoPE rescaling algorithm found by EVOLUTIONARY SEARCH guided by needle-in-a-haystack perplexity, rather than by a hand-derived rule; (3) mixed context-window training - fine-tune the weights on long sequences with rescaled RoPE while keeping the original RoPE for short contexts, so short-context performance is preserved rather than traded away.

Paper's own claim (abstract): extends LLaMA3-8B to a 128K effective context length while retaining over 98.5% of short-context performance, using only 10B training tokens - 80x fewer than Meta's approach, which (per the paper) fails to reach the target effective context length. The number to carry is the 10B-token cost with the quality retention; no inference throughput is claimed.

arXiv ID 2502.20082 verified via abs-page meta tags on 2026-10-03; title matches. CORRECTION TO THE SOURCE RECORD: it says 'arXiv preprint; no journal_ref' - wrong. OpenReview returns venue='ICML 2025 poster', venueid='ICML.cc/2025/Conference' for the exact title. venue=icml, year=2025. adoption=in-upstream-engine is about the MECHANISM being present in engines, not about this paper specifically being deployed: transformers has no distinct 'longrope2' rope_type, but the mechanism it adds - evolutionary search plus mixed-context-window training - reduces to a rescaled rope_parameters dict that transformers and vLLM already consume via the longrope path. Recorded at in-upstream-engine with that caveat rather than research-only, because the deployable artefact (a RoPE rescaling config) is consumed by shipped code. Contrast with the 'naming collision' note on longrope: this is a genuinely different method, not a v2 of the same one. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.