id
flops/ctx-claimed-vs-effective-context-table
Arithmetic intensity
This record is not an arithmetic-intensity claim; the cost side of long context lives in [[ctx-kv-cache-gib-at-claimed-context]] and [[ctx-128k-prefill-compute-cost]]. What it establishes is the input side: how much of the advertised context a model actually uses. RULER defines effective context length as the longest tested length (4K/8K/16K/32K/64K/128K) whose 13-task average score clears the threshold equal to Llama-2-7B-chat's own score at 4K, which the paper reports as 85.6 percent; passing lengths are marked by underlining in the paper's tables and restated in the Claimed/Effective columns of the repo's results table. Using one model's own short-context score as the bar is what makes the metric self-referential and comparable across model sizes: it asks 'does this model still do this task at 4K as well as a 4K model did'. AVERAGE GAP ACROSS THE MODELS MEASURED BY NVIDIA LOCALLY (claimed / effective / ratio): Yi-34B 200K/32K = 6.3x; Command-R-plus 128K/32K = 4.0x; Command-R 128K/32K = 4.0x; Qwen2-72B 128K/32K = 4.0x; Mistral-Large-2407 128K/32K = 4.0x; Mixtral-8x22B 64K/32K = 2.0x; Phi3-medium 128K/32K = 4.0x; Llama-3.1-8B 128K/32K = 4.0x; ProLong-8B 512K/32K = 16x; MegaBeam-Mistral 512K/32K = 16x; Mistral-v0.2-7B 32K/16K = 2.0x; Mistral-Nemo 128K/16K = 8x; GradientAI-Llama-3-70B-1048k 1M/16K = 64x; GradientAI-Llama-3-8B-1048k 1M/16K = 64x; InternLM2.5-7B-1M 1M/4K = 256x; GLM3-6B-128K 128K/4K = 32x; LWM-7B-Chat-1M 1M/<4K = >256x; DBRX 32K/8K = 4.0x; Qwen1.5-72B 32K/8K = 4.0x; Together-Llama-2-7B-32K 32K/4K = 8x; LongChat-7B 32K/<4K = >8x; LongAlpaca-13B 32K/<4K = >8x. MODELS THAT CLEAR 128K (no gap within the tested range): Llama-3.1-70B 128K/64K (2.0x), GPT-4-1106-preview 128K/64K (2.0x), GLM-4-9B-1M 1M/64K (>16x claimed but effective only 64K), Gemini-1.5-Pro 1M/>128K, Qwen2.5-7B-Instruct-1M 1M/>128K, Qwen3-235B-A22B 128K/>128K, Qwen3-14B 128K/>128K, Jamba-1.5-large 256K/>128K, Jamba-1.5-mini 256K/>128K. THE GAP SHRINKS FAST WITH MODEL SIZE, which is the single most actionable pattern in the table: within one family at one context claim, Llama-3.1-70B reaches 64K where Llama-3.1-8B reaches 32K; GLM-4-9B-1M reaches 64K where GLM3-6B-128K reaches 4K; Yi's own ablation holds training length and data blend fixed across 6B/9B/34B at 200K and finds effective length rises with size; Qwen3-30B-A3B (30B) reaches only 64K while Qwen3-32B (32B dense) clears 128K, and Qwen3-8B reaches 64K. So 'the context window does not work' is usually 'the context window does not work at this parameter count'. Where the gap is worst is the fine-tune-on-top-of-a-frozen-base family: LongChat-7B <4K, LongAlpaca-13B <4K, Together-32K 4K, DBRX 8K, GradientAI-1048k 16K - all fine-tuned continuations whose claimed length is several times the length they were actually trained at. Also note the two AGGREGATE metrics RULER reports, which disagree with each other and both disagree with the effective length: a linearly-increasing weighted average (inc) that assumes short contexts dominate real traffic, and a decreasing one (dec) that assumes long ones do. Llama-3.1-70B scores 89.6 avg / 85.5 wAvg-inc / 93.7 wAvg-dec, and Command-R-plus-0824 87.9 / 83.4 / 92.4 - so which aggregate you read decides whether Command-R-plus looks better or worse than Llama-3.1-70B. Effective length is the metric that survives both, because it is a threshold crossing rather than an average.
Scales with
parameter count (larger models hold effective length longer - the dominant observed effect)how the context was obtained (fine-tuning a short-context base underperforms pre-training at length, per RULER's own ablation)position-embedding scaling method, which is only half the story and is recorded separately in [[ctx-rope-scaling-method-per-model]]task category (multi-hop tracing collapses first; see the per-category table in notes)
Workarounds
Read the per-category effective length, not the single aggregate: retrieval tasks survive much longer than multi-hop tracing, so pick the benchmark that matches the workloadPrefer models trained at the target length over fine-tuned continuations of short-context bases, even at equal parameter countCap the advertised context in your own router to the tested effective length for the task class the request actually isEvaluate on RULER or MRCR yourself at the length you plan to serve, not on a vendor card: the vendor number is the claimed column
notes
PER-CATEGORY EFFECTIVE LENGTH, which is why the single number is dangerous (from the paper's Tables 13-16, arXiv 2404.06654v3): Llama-3.1-70B claimed 128K gives 64K retrieval / 32K variable-tracking / 64K aggregation / 64K QA. Llama-3.1-8B claimed 128K gives 64K / 8K / 128K / 128K - a 16x spread within one model, with variable tracking the worst category and aggregation the best. Yi-34B claimed 200K gives 64K retrieval / 16K variable tracking / >128K aggregation / 128K QA, so the widely-quoted 'Yi-34B is 200K' and the widely-quoted 'Yi-34B effective 16K' are both true and describe different tasks; the 16K figure is multi-hop tracing, the 64K figure is retrieval, and the 13-task aggregate is 32K. Qwen2-72B claimed 128K gives 64K / 32K / 64K / 64K. Command-R-plus claimed 128K gives 64K / 32K / 64K / 128K. GLM-4-9B-1M claimed 1M gives 64K / 8K / >128K / >128K. PROVENANCE WARNINGS on the numbers above, because this is a table assembled from several places: (a) RULER's table mixes NVIDIA's own runs with MODEL AUTHORS' self-reported numbers imported from their papers, marked * in the README (Jamba-1.5, FILM-7B, Qwen2.5-1M, Qwen3, EXAONE 4.0); for those rows 'effective length' means 'what the vendor said', which is exactly the quantity this record exists to distrust; (b) Gemini-1.5-Pro, GPT-4-1106-preview and LWM were run through API endpoints, so the served tokenizer, sampling and any server-side truncation are not controlled by the benchmark; (c) effective length is quantized to the 4K/8K/16K/32K/64K/128K grid, so a real 100K effective length reads as 64K - the metric can only ever understate a model that sits between rungs; (d) the threshold is Llama-2-7B-chat at 4K, an ABSOLUTE bar, so a stronger model must clear a score set by a much weaker one, and this cuts in the conservative direction; (e) RULER's own README tells users to raise max_position_embeddings in config.json to test beyond a model's declared length, so some evaluated lengths exceed the shipped config - the model's weights are unchanged but its rope settings are, which is the mechanism that makes extrapolation degrade; (f) RULER's Limitations section concedes it covers only 13 task configurations selected so that models perform well at 4K, and explicitly cannot replace realistic tasks. RELATION TO THE CLAIM COLUMN: the claimed number is read from each model's config.json ([[ctx-src-hf-configs-longctx-models]]) and does NOT always equal what vendors advertise in prose. Three concrete mismatches found: Qwen3-235B-A22B and Qwen3-8B both ship max_position_embeddings 40960 while every RULER row, model card and API listing says 128K; Qwen3-Next-80B-A3B ships 262144; GLM-4-9B-chat-1M uses the legacy key seq_length 1048576 rather than max_position_embeddings, so tooling that greps only for max_position_embeddings reads it as having no declared limit at all. And the DIRECTION of the mismatch matters: Qwen3's config is 3.1x SHORTER than the advertised 128K, so a server that trusts config.json will reject prompts the API happily accepts. LongBench, the other long-context benchmark named in the brief, is a REAL benchmark with published per-model numbers but measures task quality on 21 realistic bilingual datasets averaging 6,711 English words - it is not a context-length probe and has no 'effective length' column. It is best used as a QUALITY ceiling check alongside RULER's LENGTH ceiling, not as a substitute: a model can hold a long context (RULER passes) and still score poorly on real long-document tasks (LongBench). LongBench's own finding, from its abstract, is that 'Scaled position embedding and fine-tuning on longer sequences lead to substantial improvement on long context understanding' - which is the same causal claim as this record's training-vs-fine-tune split, stated independently.