Long-Context Windows: What Independent Benchmarks Show Versus What Vendors Claim

Over the past two years, the number printed on a language model's spec sheet as its "context window" has grown faster than almost any other metric in the field. In June 2024, Google made Gemini 1.5 Pro's 2-million-token context window generally available to all developers, after an earlier limited preview. In April 2025, Meta announced that Llama 4 Scout supports a 10-million-token context window, and OpenAI shipped GPT-4.1 with a 1,047,576-token context window in 2025. Later that year, Anthropic extended Claude Sonnet 4 to a 1-million-token context window in 2025. These are real engineering achievements: attention mechanisms, KV-cache management and positional encodings that once broke down past a few thousand tokens now process inputs the length of a long novel or an entire mid-sized codebase.

The question this piece asks is narrower and more practical: when a model is fed most of that window, does it actually use the information inside it? A context window is an architectural limit on how many tokens a model can accept, not a measurement of how well it reasons over them. Vendors report the former number prominently because it is easy to state and easy to compare. Independent researchers have spent the past two years building benchmarks that measure the latter, and the gap between the two numbers is large enough to change how the technology should be used.

The short version, supported by the benchmarks below: on simple retrieval tasks, most current frontier models hold up reasonably well through a meaningful fraction of their advertised window. On tasks that require connecting information without exact keyword overlap, tracking multiple facts at once, or reasoning over intermediate state rather than recalling isolated sentences, performance degrades much earlier — often by an order of magnitude sooner than the advertised limit. None of the four vendor claims quoted above has yet been evaluated at its full advertised length by an independent, published benchmark using the methodologies described here; the sections below explain what has actually been measured, and what has not.

What the "needle in a haystack" test actually checks

The most widely cited long-context evaluation for several years was the "needle in a haystack" (NIAH) test: insert one sentence of unrelated information (the needle) into a long, irrelevant document (the haystack) and ask the model to retrieve it. The test is cheap to run and produces clean pass/fail heatmaps, so it became a standard slide in model release announcements. But it measures only literal, single-fact retrieval under close-to-ideal conditions — the needle sentence is usually distinctive and the question closely echoes its wording. Researchers evaluating long-context claims more rigorously have converged on the view that NIAH is necessary but far from sufficient, because a model can score close to 100% on it while failing at tasks that require aggregating several pieces of information or making an inferential link rather than a lexical match.

RULER: separating claimed length from effective length

NVIDIA's RULER benchmark, introduced by Hsieh et al. at COLM 2024, was built specifically to go beyond NIAH. It adds multi-hop tracing, variable tracking, aggregation and multi-document question answering tasks, all synthetically generated so that a model cannot lean on memorized world knowledge. The paper defines a model's "effective context length" as the longest length at which it still clears a quality bar relative to its own short-context performance, distinct from the "claimed" length printed in its documentation.

The gap the paper reports is substantial. Testing seventeen open- and closed-weight models across thirteen synthetic tasks, the authors found that despite near-perfect scores on the vanilla NIAH task, most models showed large accuracy drops on the fuller RULER suite as length increased; by the paper's own count, only about half of the seventeen models evaluated maintained satisfactory performance at 32,000 tokens. GPT-4, Command-R (35B), Yi-34B and Mixtral (8x7B) were among the models that cleared that bar, each detailed in the table below, while several others claiming context windows of 128K tokens or more fell short of it. On the GPT-4 (1106-preview) results reported in the paper and in NVIDIA's public RULER leaderboard, accuracy fell from 96.6% at 4K tokens to 81.2% at its 128K claimed length, giving it an effective length of roughly 64K — half its advertised window. Mistral 7B, which claims 32K, scored 93.6% at 4K but only 13.8% at 128K, with an effective length the paper puts at 16K. The same leaderboard's later entries show some later models closing the gap: Gemini 1.5 Pro is reported with scores staying above 94% out to 128K tokens, and Llama 3.1 70B is reported with an effective length around 64K, moving from 96.5% at 4K to 66.6% at 128K. RULER has not published results for Llama 4 Scout's 10-million-token window, GPT-4.1's 1-million-token window, or Claude Sonnet's 1-million-token window as of this writing — its leaderboard remains dominated by 2024-era models. The NoLiMa results below add independent figures for two of those three claims, gathered through a different methodology.

Read as a set, the table makes the central point of the RULER paper concrete: the number a vendor advertises is an upper bound on what the architecture will accept, not a claim about accuracy at that length, and the two should not be treated as interchangeable when choosing a model for a task that depends on using most of the input.

NoLiMa: removing the keyword shortcut

A second limitation of NIAH-style tests is that the "needle" sentence and the question about it typically share vocabulary, letting a model succeed through something closer to string matching than genuine reasoning over the context. Modarressi et al.'s NoLiMa benchmark, accepted at ICML 2025, was designed specifically to remove that shortcut: each needle is written so that the question that later probes it shares minimal literal vocabulary with the needle, forcing the model to draw a latent, semantic connection rather than pattern-match a phrase.

The effect on measured performance is sharper than in RULER. Evaluating thirteen models that claim context support of 128,000 tokens or more, the NoLiMa authors found that at 32,000 tokens, eleven of the thirteen models dropped below half of their own short-context (under 1,000-token) baseline score. GPT-4o's own results in the paper illustrate the pattern: its score fell from a near-perfect 99.3% baseline to 69.7% once the literal-matching shortcut was removed and the context extended. That is a materially larger drop than GPT-4's RULER curve above, which is expected — NoLiMa is testing a harder property (semantic association across distance) rather than plain retrieval, and the two benchmarks are not directly comparable in absolute terms, but both point the same direction: real-world tasks, where the wording of a query rarely matches the source document verbatim, are likely to sit closer to the NoLiMa curve than to the friendlier NIAH heatmaps vendors typically publish.

Since the paper's initial release, its authors have extended the same evaluation to newer, longer-context models, and the benchmark's published repository shows the pattern holding. GPT-4.1, which advertises a 1,047,576-token window, falls to roughly 65% accuracy at 128,000 tokens — a small fraction of its claimed length. Llama 4 Scout, which advertises 10 million tokens, falls further and sooner: to roughly 22% at just 32,000 tokens, with the smaller Llama 4 Maverick (1 million tokens claimed) at roughly 25% on the same test. Claude Sonnet's 1-million-token window has not yet been tested by NoLiMa; its published results still cover the earlier, 200,000-token Claude 3.5 Sonnet, which falls to roughly 30% at 32,000 tokens. None of these newer figures come close to the vendors' full advertised lengths, but they are the closest thing to independent, semantic-matching evidence that currently exists for two of the three most recent claims described at the start of this piece.

Lost in the middle: position matters even inside the "effective" range

A separate and earlier finding complicates things further even within a model's effective range. Liu et al., published in the Transactions of the Association for Computational Linguistics in 2024, tested how language models use long contexts in multi-document question answering and key-value retrieval and found a consistent U-shaped performance curve: models retrieve information most reliably when it sits near the beginning or the end of the input, and accuracy drops noticeably when the same information is placed in the middle of a long context — a pattern that held even for models explicitly built and marketed for long-context use. The practical implication is that "effective context length" as reported by RULER is itself an average over positions; a document's most important fact, if placed in the middle of a 50,000-token input, may be retrieved less reliably than the aggregate number for that length would suggest.

Beyond retrieval: multi-step reasoning fails earlier still

All three benchmarks above are fundamentally retrieval or matching tasks — even NoLiMa, which requires an inferential hop, asks the model to locate a single relevant fact. A newer strand of work asks whether models can maintain and update state across a long context, which is a different and harder demand than finding a fact within one. Pan and Williams, in a 2025 preprint currently under review, introduce a benchmark called Verbose ListOps (VLO) that embeds deterministic list-processing computations inside narrative text, so that answering correctly requires tracking a chain of intermediate results rather than recalling a single stated fact. Their reported finding is stark: models that solve the underlying computation with close to 100% accuracy when it is presented in raw, unadorned form collapse well before 10,000 tokens once the same computation is embedded in narrative camouflage. Because this is a preprint rather than a peer-reviewed publication, its specific numbers should be treated as provisional pending review — but the direction of the result is consistent with RULER's multi-hop tracing tasks and NoLiMa's inference requirement, all three converging on the same conclusion: tasks that require doing something with information across a long context fail earlier and more sharply than tasks that only require finding it.

What to check before trusting a context-window number

None of this means the advertised windows are false — a 1-million or 10-million-token window is a genuine statement about how many tokens the model's architecture and serving stack will accept as input, and that ceiling is itself useful for some tasks, such as ensuring a large document simply fits without chunking. What it does not establish, on its own, is that information anywhere in that window will be found, connected or reasoned over as reliably as information near the front. Three practical checks follow from the benchmarks above:

• Ask whether a claimed context length has an independently measured effective length attached to it. As of this writing, RULER results exist for GPT-4-class, Gemini 1.5, Llama 3.1, Mixtral, Mistral, Command-R and Yi-class models, and NoLiMa has since added GPT-4.1 and the Llama 4 family; both benchmarks stop far short of the millions-of-tokens figures themselves, and neither has published results for Claude Sonnet's 1-million-token window, so that specific claim currently rests on vendor-reported NIAH-style results rather than independent multi-task measurement.
• Treat needle-in-a-haystack heatmaps in a release announcement as a floor, not a summary. A model can score near 100% on NIAH and still fail the semantic-inference tasks in NoLiMa or the multi-hop tasks in RULER at the same length.
• For tasks where the critical information could plausibly sit mid-document — long contracts, long transcripts, large codebases — the positional effect documented in Lost in the Middle means placement inside the prompt is not neutral; where possible, testing with the key fact in different positions is more informative than a single pass.

The broader pattern across RULER, NoLiMa, Lost in the Middle and the VLO preprint is that "context window" and "usable context" are two different quantities that happen to share a unit of measurement. The first has grown by roughly three orders of magnitude in under two years, from tens of thousands of tokens to the millions now advertised by Anthropic, Google, Meta and OpenAI. The second has grown too, but more slowly and unevenly across tasks, and independent benchmarking is the only way to know, for a specific model and a specific kind of task, how much of the advertised number is actually load-bearing.