LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

LLMs do not generalise to contexts longer than their training length, and the usual fixes all require fine-tuning, which is expensive and needs data the model owner may not have.

Build BI-LEVEL attention information at inference, with no weight changes. GROUPED attention pools distant tokens into groups (and selects the most relevant groups per query) to capture long-range dependency without O(n^2); NEIGHBOR attention handles nearby tokens within a specified local range with the original exact self-attention. Both are computed from the existing model's own attention mechanism with a small code modification, so the weights are untouched.

NOT RECORDED AS A NUMBER. The abstract's claim is qualitative and capability-shaped: SelfExtend effectively extends existing LLMs' context window length, evaluated across multiple benchmarks, with 'minor code modification' and no fine-tuning. No multiple, no baseline, no hardware. The deployment-relevant property is architectural: no retraining and no architecture change means it applies to an already-trained checkpoint.

arXiv ID 2401.01325 verified via abs-page citation_title/citation_date on 2026-10-03; the arXiv page's citation_title is 'LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning', matching the record name exactly. Venue confirmed twice: arXiv comment 'ICML2024 Spotlight' and OpenReview venue='ICML 2024', venueid='ICML.cc/2024/Conference'. venue=icml, year=2024. adoption=research-only verified by absence: no selfextend or LongLM path in vLLM's 8562-file tree, none in SGLang's 11562-file tree, and no corresponding cache layer in transformers cache_utils.py. The efficient ecosystem instead adopted the RoPE-rescaling family of context extension - yarn, longrope, position-interpolation, all recorded separately here and all in-upstream-engine - because a rescaling config is consumed by existing kernels whereas grouped+neighbor attention needs new attention code per head. That contrast is the useful lesson in this record. The grouping mechanism does resemble what H2O/Quest do, but at prefill rather than at KV eviction. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.