A Length-Extrapolatable Transformer

A transformer trained on short texts degrades sharply when evaluated on longer ones, and the usual framing (interpolate vs extrapolate positions) does not explain WHY: it lacks a quantity that predicts which variant will survive length extrapolation.

Two changes, both about the frequency spectrum rather than about position indices. (1) Introduce a RELATIVE position embedding to explicitly maximise 'attention resolution', defined in the paper as a log-base exponential of the smallest discrete positional difference the model can distinguish. (2) Use blockwise causal attention at inference for better resolution. The widely used name 'NTK-aware scaled RoPE' for this work comes from the associated blog post, not from the paper's own title.

NOT RECORDED. The abstract's quantitative content is empirical language-modelling comparison across interpolation and extrapolation settings, with no speedup, throughput or context-length figure carrying a baseline. This paper is a diagnostic and a method, not a systems result - the durable contribution is the 'attention resolution' indicator.

arXiv ID 2212.10554 verified via abs-page meta tags on 2026-10-03; title 'A Length-Extrapolatable Transformer' matches the record name. venue=arxiv-preprint confirmed as the right call: the arXiv comment field reads only '9 pages', OpenReview returns no record for this title (an OpenReview search surfaces only later papers that CITE or reuse it, e.g. LEDiT at NeurIPS 2025), and Semantic Scholar was unavailable (429). Naming caution preserved from the source record and worth restating because it is a genuine trap: this paper is NOT the source of the 'NTK-aware scaled RoPE' technique as deployed - the deployed technique descends from the associated blog post, and the shipped engine code names it separately. adoption=in-upstream-engine therefore refers to the RELATIVE-PE / NTK-scaling family being present in shipped code (transformers ROPE_INIT_FUNCTIONS has 'dynamic' -> _compute_dynamic_ntk_parameters; vLLM ships DynamicNTKScalingRotaryEmbedding and DynamicNTKAlphaRotaryEmbedding), not to this specific paper being implemented. Deliberately flagged so the record is not read as saying 'this paper is what vLLM ships'. VENUE CORRECTION + VENUE_TRACK 2026-10-04 (paper-fields pass, agent w3p): venue 'arxiv-preprint' -> 'acl', venue_track 'main'. EVIDENCE: the ACL Anthology bib index (anthology.bib, 131650 entries, fetched 2026-10-04) contains this exact title as anthology id 2023.acl-long.816 with booktitle 'Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)'. That is a proceedings record, so 'no venue found' was an under-claim, not a fact. BOTH VALUES RETAINED HERE PER THE REPO CONTRACT: the prior value was 'arxiv-preprint'. status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2212.10554). 2026-10-10: arXiv API 2212.10554 (axc.py: http://export.arxiv.org/api/query?id_list=2212.10554&max_results=1) comment is null. ACL Anthology 2023.acl-long.816 confirms ACL 2023 (note: year 2023, while record carries 2022). Source pap-attn-ntk-aware-scaled-rope notes cite arXiv preprint vs ACL Anthology 2023.acl-long.816. Sources disagree; status remains contested.