SWE-bench Verified's Scores Looked Great. OpenAI Says They're Contaminated

For two years, a single number stood in for how good an AI model was at software engineering: its score on SWE-bench Verified. The benchmark gave a model a real GitHub issue and a snapshot of the codebase it belonged to, and asked it to produce a patch. In February 2026, OpenAI, the company that co-published the "Verified" version of the test, said it would stop reporting scores on it, citing contamination. The reversal did not come alone. An academic study and a new benchmark built by Scale AI had already shown, with numbers, how much of the benchmark's apparent progress was not what it looked like.

The original benchmark, introduced by Carlos Jimenez and colleagues at Princeton in a paper accepted as an oral presentation at ICLR 2024, drew 2,294 task instances from 12 popular Python repositories. Each task paired a real issue report with the pull request that had actually resolved it on GitHub. A model was handed the issue text and the repository at the commit before the fix, and had to produce a patch of its own. Success was graded mechanically: the repository's own test suite was run inside a Docker container, and a task counted as "resolved" only if the specific tests that failed before the real fix — the "FAIL_TO_PASS" tests — now passed, without breaking tests that had passed before. At the time of that 2024 paper, the best-performing model, Claude 2, solved a mere 1.96% of the issues.

That low ceiling, and the benchmark's realism, is what made it attractive as a yardstick. But the original set also had problems: some issues were underspecified, some test patches were wrong, and some tasks were effectively unsolvable from the information given. In August 2024, OpenAI's preparedness team worked with the original authors to fix this. Ninety-three experienced Python developers manually reviewed 1,699 randomly sampled tasks from the SWE-bench test set, with three separate annotators checking each one for a clear problem description, a correct test patch, and a solvable scope, as described in OpenAI's announcement. What survived became SWE-bench Verified: 500 tasks.

The climb that made the leaderboard

Over the following year, frontier labs reported steadily rising scores on that 500-task set. Anthropic's announcement of Claude Opus 4.1 on 5 August 2025 put the model at 74.5% on SWE-bench Verified, measured — the company specified — across the full 500 problems, up from 72.5% for the preceding Opus 4. Two days later, OpenAI's GPT-5 system card reported 74.9% on SWE-bench Verified, up from o3's 69.1%, but measured on a fixed subset of 477 of the 500 tasks that OpenAI says it had validated on its own infrastructure — a detail that makes the two companies' numbers not directly comparable, even though both are routinely quoted side by side as if they were.

Those figures already hint at a problem that goes beyond task counts. A benchmark built from public GitHub repositories, with its 500 verified issues and correct patches posted openly for reproducibility, is also a benchmark that language models can have read during training. Whether that exposure helps a model reason about similar bugs, or lets it simply recall the fix, is not something a single pass/fail score can distinguish.

Testing for memorization directly

That is the question Shanchao Liang, Spandan Garg and Roshanak Zilouchian Moghaddam set out to answer in "The SWE-Bench Illusion," first posted to arXiv in June 2025 and revised through December 2025. Rather than scoring full patches, they gave ten model checkpoints — two dated snapshots of OpenAI's GPT-4o, plus GPT-4.1, o3, o3-mini and o4-mini, and Anthropic's Claude 3.5 Sonnet, 3.7 Sonnet, 4.0 Sonnet and 4.0 Opus — two diagnostic tasks that require recognition rather than coding: naming the file path containing a bug from the issue text alone, and reproducing the exact wording of a target function.

On SWE-bench Verified and the full SWE-bench set, models identified the correct buggy file path with up to 76% accuracy using only the issue description, with no access to the repository. On equivalent tasks drawn from repositories outside the SWE-bench ecosystem, such as pandas and PyTorch, accuracy fell to as low as 53%, even though those repositories are at least as popular and likely just as present in training data. A second diagnostic showed the same pattern: models reproduced target functions with up to 35.0% consecutive 5-gram accuracy on SWE-bench tasks, against up to 18.2% on instances outside that ecosystem. The authors describe this gap as evidence that gains on the benchmark reflect memorization of specific SWE-bench instances rather than a general ability to localize or reproduce code.

A benchmark built to resist recall

Scale AI reached a similar conclusion from a different angle. On 19 September 2025, it published SWE-bench Pro, a benchmark of 1,865 tasks across 41 repositories, deliberately built so the code could not simply have been memorized. Of those repositories, 11 are public ones released under strong copyleft licenses such as the GPL, whose "viral" terms and legal complexity Scale says make them unlikely to appear in commercial training corpora; 18 are entirely private commercial codebases that Scale purchased from companies; and 12 form a held-out set. The 1,865 tasks split into 731 public, 276 commercial and 858 held-out instances.

The scores collapsed. Scale's post states that the best-performing models, OpenAI's GPT-5 and Anthropic's Claude Opus 4.1, scored only 23.3% and 23.1% respectively on SWE-bench Pro overall — against the 74%-plus each had posted on SWE-bench Verified. Restricted to the private commercial subset, which the models could not plausibly have seen in any form, Claude Opus 4.1's resolution rate fell further, to 17.8%, and GPT-5's to 14.9%. Older or smaller models fared far worse still: GPT-4o resolved 4.9% of SWE-bench Pro tasks, and DeepSeek Qwen-3 32B resolved 3.4%. Scale frames the gap between Pro and Verified as a measure of how much of the Verified score was earned on code the models had plausibly already encountered.

OpenAI's own retraction

The clearest acknowledgment came from the benchmark's own co-publisher. In a February 2026 post, "Why We No Longer Evaluate SWE-bench Verified," OpenAI gave two reasons, not one. The first was contamination: the company had GPT-5 probe GPT-5.2-Chat, Claude Opus 4.5 and Gemini 3 Flash Preview on SWE-bench Verified questions, and recovered near-verbatim patches and task-specific details the models could only have gotten from training exposure — and separately noted that scores had risen from 74.9% to 80.9% over the preceding six months, a pace it treats as a sign of leakage rather than genuine capability gains. The second was task quality: OpenAI had experienced engineers independently re-review 138 problems that its o3 model consistently failed, and found that 59.4% had material test-design flaws — 35.5% enforced one specific implementation when other correct solutions would be rejected, and 18.8% checked for functionality the issue never asked for. Taken together, OpenAI said, a model's remaining failures increasingly reflect dataset problems rather than coding skill, and its remaining successes increasingly reflect prior exposure rather than reasoning. The company said it now recommends SWE-bench Pro as the standard for reporting frontier coding results.

What the numbers can responsibly tell a reader

None of this means SWE-bench Verified ever measured nothing. The original filtering process, with 93 developers and triple review, removed many genuinely broken tasks, and a model that could not pass any SWE-bench task at all would not be capable of much. What the Illusion study, SWE-bench Pro and OpenAI's own audit show is narrower and more useful: that a score's meaning depends entirely on whether the evaluator can rule out prior exposure to the exact task and rule out flawed tests rewarding one arbitrary implementation, and that two models' scores on "SWE-bench Verified" are only comparable if measured on the same count of tasks under the same harness — 500 versus 477 is not a rounding difference, it is a different test.

For anyone using these numbers to choose a model or to judge the field's progress, the practical checklist is now explicit, courtesy of the people who built and then retired the benchmark: ask which exact task set and harness produced a reported score, ask how recently it was measured, and treat any benchmark whose answer key has been public for more than a few months with the same skepticism OpenAI now applies to its own earlier evaluations. SWE-bench Pro's commercial subset, kept private specifically so it cannot leak, is the closest thing currently available to an answer that resists that problem — and even there, the best models solve fewer than one task in five.