inference/research
verifiedhigh · 0.85updated 2026-10-04

Why the gap exists: which RoPE/context-extension method each long-context model actually used, and trained-at-shorter-than-licensed

compute bound

Fields

id
flops/ctx-rope-scaling-method-per-model
Class
prefill_attention
Bound by
compute
Arithmetic intensity
Position-embedding scaling does not change attention FLOPs or KV bytes at all - it is a pure index remap on the rotary phase. AI is unchanged; what changes is whether the model's learned function of relative distance still matches the distances it is now being asked about. That is why this record is filed under prefill_attention as a QUALIFIER on that cost rather than a cost of its own: a model whose rope settings were extended without matching training sees the same prefill bill as one that was properly trained, and produces worse answers for it. The arithmetic that does move: NTK-aware scaling and YaRN change the base theta, so effective wavelength coverage per dimension changes, and PI/LongRoPE change per-dimension position division, which resamples position indices rather than rotating them - neither adds flops, and neither adds a single byte to the KV cache, which stores post-projection K and V with no positional information in them at all. The serving consequence is therefore entirely on the accuracy side of [[ctx-claimed-vs-effective-context-table]], not the cost side.
Scales with
the extension factor (rope_scaling.factor: 8x for Llama 3.1, 40x for DeepSeek V2/V3, 32x for Kimi-K2)the training length actually used versus the advertised length (original_max_position_embeddings vs max_position_embeddings)base theta (500000 for Llama 3.1, 1e6 for Qwen2.5/Qwen3, 1e7 for Qwen3-Next and Yi, 10000 for DeepSeek V2/V3/V3.2, 50000 for Kimi-K2, 8e6 for Command-R-plus)
Workarounds
Read original_max_position_embeddings against max_position_embeddings to see the honest extension factor: 8192 -> 131072 is 16x claimed but only 8x of rope interpolation on Llama 3.1For models that ship rope_scaling: null but a large claimed length (Qwen3 at 40960, Qwen3-Next at 262144, Qwen2.5 at 32768, Mistral Large at 131072, Command-R-plus at 131072, GLM-4.5 at 131072, Yi at 200000), the length was obtained by pre-training at length, not by index remapping - so extrapolate these with the training method, not with PISet the server's rope parameters from config.json rather than letting a default 8x PI fire on any model whose rope_scaling is nullWhere a vendor claims 1M on a base whose rope_scaling is null, expect the RULER-style degradation to appear only in vendor self-reports and not in the config
notes
PER-MODEL SCALING METHOD, read from config.json on 2026-10-04 ([[ctx-src-hf-configs-longctx-models]]). NAKED RoPE WITH NO INTERPOLATION (rope_scaling: null): Qwen2.5-7B-Instruct (32768, theta 1e6), Qwen2.5-72B-Instruct (32768, theta 1e6), Qwen2.5-7B-Instruct-1M (1010000, theta 1e10), Qwen3-8B (40960, theta 1e6), Qwen3-235B-A22B (40960, theta 1e6), Qwen3-Next-80B-A3B (262144, theta 1e7), Mistral-Large-2411 (131072, theta 1e6), Command-R-plus-0824 (131072, theta 8e6), Yi-34B-200K (200000, theta 1e10), GLM-4.5 (131072, theta 1e6). For this group the only defense against extrapolation failure is a large base theta plus pre-training at the target length - which is exactly why Qwen3's 40960 config with theta 1e6 behaves well at 128K when RULER-tested (all four Qwen3 rows clear 128K) while a smaller model with the same null rope_scaling does not. LIN 3-STYLE RoPE SCALING (Meta's own method, shipped as rope_type: llama3): all three Llama 3.1 sizes - 8B, 70B, 405B - carry {factor 8.0, low_freq_factor 1.0, high_freq_factor 4.0, original_max_position_embeddings 8192} with theta 500000. This is frequency-splitting, not uniform interpolation: high-frequency dimensions get high_freq_factor 4.0 (preserved almost verbatim), low-frequency dimensions get low_freq_factor 1.0 (fully stretched 8x). The arithmetic worth writing down because it is the mechanism behind Llama 3.1's relatively good showing: 131072/8192 = 16x claimed growth but only 8x interpolation on the stretched dimensions and 4x on the preserved ones, so no dimension is stretched more than 8x - and 8x is inside the range where the model was still trained to discriminate relative distance. YaRN (type: yarn): DeepSeek-V2 {factor 40, mscale 0.707, mscale_all_dim 0.707, beta_fast 32, beta_slow 1, original_max_position_embeddings 4096, theta 10000}, DeepSeek-V3 and V3.2 {identical except mscale 1.0}, Kimi-K2 {factor 32.0, mscale 1.0, beta_fast 1.0, beta_slow 1.0, original 4096, theta 50000}. DeepSeek's 40x is by far the most aggressive interpolation in this set - 163840/4096 = 40x exactly matches factor 40 - and the compensating mscale term differs between V2 (0.707, i.e. attention logits rescaled by 1/sqrt(2)) and V3 (1.0, no rescale). CHATGLM DOES NOT USE RoPE AS CONFIGURED: glm-4-9b-chat-1m sets rope_ratio 10000 with original_rope true, and chatglm3-6b-128K inherits that family; the 1M and 128K lengths come from the ChatGLM position-encoding scheme rather than from any of these interpolations, so nothing here transfers to them. THE THREE THINGS THAT PRODUCE THE GAP, in causal order, and the repo should treat them as separate failure modes: (1) POSITION INTERPOLATION compresses the index space, so a distance the model never saw during training now maps onto one it did see - the model is not extrapolating at all, it is interpolating over a rescaled grid, and the rescaling factor is the claimed-length ratio in every case above; (2) RoPE SCALING variants (NTK-aware per arXiv 2212.10554, YaRN per arXiv 2309.00071, LongRoPE per arXiv 2402.13753) each choose WHICH dimensions to stretch, trading the high frequencies that carry local structure against the low frequencies that carry long-range structure - YaRN's beta_fast/beta_slow ramp interpolates the two and applies an mscale correction, and NTK-aware scaling changes the base itself rather than the positions; (3) TRAINED-AT-SHORTER-THAN-LICENSED, which is the case that no scaling method fixes and which the config makes visible: Llama 3.1 trained at 8192 and served at 131072, DeepSeek V2/V3 at 4096 served at 163840 (40x), Kimi-K2 at 4096 served at 131072 (32x). A fourth mechanism sits alongside these and is not a rope method at all: POSITIONAL SKIP-WISE TRAINING (PoSE, arXiv 2309.10400) simulates long inputs inside a short training window by giving each chunk of a split context a distinct position-index offset, and the paper reports extending LLaMA to 128K tokens using a 2K training context window, stating it is compatible with all RoPE-based LLMs and all position interpolation strategies. That is the cheapest way to see mechanism (3) directly - a model can reach a 128K license having never once run a 128K activation. RULER's own ablation adds the uncomfortable result that 'training on longer sequences does not always lead to better performance on Ruler', so mechanism (3) is necessary but not sufficient; the remaining variable is data blend and task mix. CORRECTION TO A COMMON READING: it is not enough to ask which scaling method a model used, because the vendors with the best RULER numbers (Qwen3, Qwen2.5-1M) used NO scaling at all - they raised the base theta and trained at length. Method choice is a poor predictor of effective length on its own; training-at-length is the better one.

Notes

PER-MODEL SCALING METHOD, read from config.json on 2026-10-04 ([[ctx-src-hf-configs-longctx-models]]). NAKED RoPE WITH NO INTERPOLATION (rope_scaling: null): Qwen2.5-7B-Instruct (32768, theta 1e6), Qwen2.5-72B-Instruct (32768, theta 1e6), Qwen2.5-7B-Instruct-1M (1010000, theta 1e10), Qwen3-8B (40960, theta 1e6), Qwen3-235B-A22B (40960, theta 1e6), Qwen3-Next-80B-A3B (262144, theta 1e7), Mistral-Large-2411 (131072, theta 1e6), Command-R-plus-0824 (131072, theta 8e6), Yi-34B-200K (200000, theta 1e10), GLM-4.5 (131072, theta 1e6). For this group the only defense against extrapolation failure is a large base theta plus pre-training at the target length - which is exactly why Qwen3's 40960 config with theta 1e6 behaves well at 128K when RULER-tested (all four Qwen3 rows clear 128K) while a smaller model with the same null rope_scaling does not. LIN 3-STYLE RoPE SCALING (Meta's own method, shipped as rope_type: llama3): all three Llama 3.1 sizes - 8B, 70B, 405B - carry {factor 8.0, low_freq_factor 1.0, high_freq_factor 4.0, original_max_position_embeddings 8192} with theta 500000. This is frequency-splitting, not uniform interpolation: high-frequency dimensions get high_freq_factor 4.0 (preserved almost verbatim), low-frequency dimensions get low_freq_factor 1.0 (fully stretched 8x). The arithmetic worth writing down because it is the mechanism behind Llama 3.1's relatively good showing: 131072/8192 = 16x claimed growth but only 8x interpolation on the stretched dimensions and 4x on the preserved ones, so no dimension is stretched more than 8x - and 8x is inside the range where the model was still trained to discriminate relative distance. YaRN (type: yarn): DeepSeek-V2 {factor 40, mscale 0.707, mscale_all_dim 0.707, beta_fast 32, beta_slow 1, original_max_position_embeddings 4096, theta 10000}, DeepSeek-V3 and V3.2 {identical except mscale 1.0}, Kimi-K2 {factor 32.0, mscale 1.0, beta_fast 1.0, beta_slow 1.0, original 4096, theta 50000}. DeepSeek's 40x is by far the most aggressive interpolation in this set - 163840/4096 = 40x exactly matches factor 40 - and the compensating mscale term differs between V2 (0.707, i.e. attention logits rescaled by 1/sqrt(2)) and V3 (1.0, no rescale). CHATGLM DOES NOT USE RoPE AS CONFIGURED: glm-4-9b-chat-1m sets rope_ratio 10000 with original_rope true, and chatglm3-6b-128K inherits that family; the 1M and 128K lengths come from the ChatGLM position-encoding scheme rather than from any of these interpolations, so nothing here transfers to them. THE THREE THINGS THAT PRODUCE THE GAP, in causal order, and the repo should treat them as separate failure modes: (1) POSITION INTERPOLATION compresses the index space, so a distance the model never saw during training now maps onto one it did see - the model is not extrapolating at all, it is interpolating over a rescaled grid, and the rescaling factor is the claimed-length ratio in every case above; (2) RoPE SCALING variants (NTK-aware per arXiv 2212.10554, YaRN per arXiv 2309.00071, LongRoPE per arXiv 2402.13753) each choose WHICH dimensions to stretch, trading the high frequencies that carry local structure against the low frequencies that carry long-range structure - YaRN's beta_fast/beta_slow ramp interpolates the two and applies an mscale correction, and NTK-aware scaling changes the base itself rather than the positions; (3) TRAINED-AT-SHORTER-THAN-LICENSED, which is the case that no scaling method fixes and which the config makes visible: Llama 3.1 trained at 8192 and served at 131072, DeepSeek V2/V3 at 4096 served at 163840 (40x), Kimi-K2 at 4096 served at 131072 (32x). A fourth mechanism sits alongside these and is not a rope method at all: POSITIONAL SKIP-WISE TRAINING (PoSE, arXiv 2309.10400) simulates long inputs inside a short training window by giving each chunk of a split context a distinct position-index offset, and the paper reports extending LLaMA to 128K tokens using a 2K training context window, stating it is compatible with all RoPE-based LLMs and all position interpolation strategies. That is the cheapest way to see mechanism (3) directly - a model can reach a 128K license having never once run a 128K activation. RULER's own ablation adds the uncomfortable result that 'training on longer sequences does not always lead to better performance on Ruler', so mechanism (3) is necessary but not sufficient; the remaining variable is data blend and task mix. CORRECTION TO A COMMON READING: it is not enough to ask which scaling method a model used, because the vendors with the best RULER numbers (Qwen3, Qwen2.5-1M) used NO scaling at all - they raised the base theta and trained at length. Method choice is a poor predictor of effective length on its own; training-at-length is the better one.

Cross-references

This record references no other records — it is a leaf. It is cited by 0.