id
gotchas/ctx-gotcha-claimed-vs-effective-context-gap
Concepts
long-contextrulerlongbencheffective-context-lengthrope-scalingposition-interpolationmodel-evaluationyi-34bllama-3-1command-rgradientai-1048kinternlm2-5
Symptom
A model loads, accepts a long prompt, and returns fluent text that is wrong. No error, no truncation warning, no OOM - just a confidently incorrect answer about something that was in the prompt. The specific figures: Yi-34B advertises 200K and its RULER aggregate effective length is 32K (6.3x), and 16K on multi-hop tracing; Command-R-plus, Qwen2-72B, Mistral-Large-2407, Phi3-medium and Llama-3.1-8B all advertise 128K and stop at 32K (4x); GradientAI's Llama-3-70B-1048k advertises 1M and stops at 16K (64x); InternLM2.5-7B-1M advertises 1M and stops at 4K (256x); GLM3-6B-128K advertises 128K and stops at 4K (32x); LWM-7B-Chat-1M, LongChat-7B, LongAlpaca-13B and Together-Llama-2-7B-32K all stop below 4K. The failure is silent because the model is still a competent language model at every length - it just stops attending to the far end of the context.
Root cause
Three separable causes, and mistaking one for another leads to the wrong fix. (1) POSITION INTERPOLATION: the rope_scaling factor in config.json equals the claimed-length ratio in every case - Llama 3.1 interpolates 8x while claiming 16x growth, DeepSeek V2/V3 interpolate 40x while claiming exactly 40x, Kimi-K2 interpolates 32x - so distances the model never saw in training are remapped onto ones it did, and it is interpolating rather than extrapolating. (2) TRAINED-AT-SHORTER-THAN-LICENSED: Llama 3.1's original_max_position_embeddings is 8192 against a 131072 limit, DeepSeek V2/V3's is 4096 against 163840, and PoSE (arXiv 2309.10400) shows a 128K license being reached from a 2K training window by positional skip-wise training alone - so the license can be granted without a long activation ever existing. (3) TASK-COMPLEXITY, not length: RULER's aggregate hides up to a 16x within-model spread, with Llama-3.1-8B scoring 128K-effective on aggregation and 8K-effective on variable tracking. That last one is the trap - the same model on the same prompt length is fine or useless depending on whether the task is retrieval or multi-hop tracing.
Workaround
Evaluate on RULER (or MRCR) at the length you intend to serve and the task class your workload actually is, and cap the advertised limit in your own router at the measured effective length for that class. Prefer models that were PRE-TRAINED at the target length over fine-tuned continuations of short-context bases - RULER's own ablation shows training longer does not always help, but the gap is worst in the fine-tune-on-top family (LongChat, LongAlpaca, DBRX, the 1048k GradientAI models). Note the countervailing finding: larger models hold effective length longer within a family, so swapping to a bigger sibling is often the cheapest fix - Llama-3.1-70B reaches 64K where 8B reaches 32K, and Yi's own 6B/9B/34B ablation at fixed 200K training length shows the same trend.
notes
This is the umbrella record for the gap; the per-model numbers live in [[ctx-claimed-vs-effective-context-table]] and the scaling methods in [[ctx-rope-scaling-method-per-model]]. Three measurement caveats that change how the numbers should be quoted. (a) RULER's 'effective length' is quantized to a 4K/8K/16K/32K/64K/128K grid, so it can only ever UNDERSTATE a model whose true effective length falls between rungs - a real 100K reads as 64K. (b) RULER's results table mixes NVIDIA's own runs with results the MODEL AUTHORS reported in their own papers, marked with an asterisk in the README; for those rows the 'effective length' is a vendor claim, which is precisely the quantity this record exists to distrust. (c) The threshold is Llama-2-7B-chat's own score at 4K (reported as 85.6 percent), an absolute bar, so the metric is conservative. Also note that LongBench is a real benchmark with published per-model numbers but is not a length probe - it measures task quality on 21 realistic bilingual datasets averaging 6,711 English words and has no effective-length column, so it is a quality ceiling check to run ALONGSIDE RULER, not a substitute. Sources: [[ctx-src-ruler-github-results-table]], [[ctx-src-ruler-paper-yi-ablation]], [[mkt-ruler-long-context-benchmark]], [[ctx-src-pose-positional-skip-training]].