SWE-bench Verified Is Saturated: What the Score Actually Measures Now

When an AI lab announces a new coding model, the number that follows is almost always a score on SWE-bench Verified: the percentage of real GitHub issues the model can resolve end to end, unassisted. The benchmark has become the closest thing the field has to a shared unit of progress, cited in model cards, press releases and research papers alike. It is also, according to the people who built it and the people who have since audited it, no longer measuring what most of those announcements imply it measures.

SWE-bench was introduced by researchers at Princeton and presented as an oral paper at the 2024 International Conference on Learning Representations. The original dataset drew 2,294 issue-and-pull-request pairs from twelve popular Python repositories: a model is given a real bug report and the surrounding codebase, and is judged by whether its patch makes the repository's own test suite pass. SWE-bench Verified, a 500-task human-screened subset released later to remove ambiguous or unsolvable issues, became the version most labs actually report. The appeal is obvious: unlike multiple-choice knowledge tests, it asks a model to do something close to a day-one job of software engineering, and it grades the output against tests that were already sitting in the codebase, not against a rubric a lab wrote itself.

The trouble is what "passing the test suite" turns out to mean in practice, and two independent research teams have now measured it directly.

What "solved" turns out to mean

A 2024 study by researchers including Reem Aleithan and Song Wang, published as SWE-Bench+, manually screened the patches that SWE-Agent paired with GPT-4 had produced on issues the benchmark marked as resolved. They found that 32.67% of the "successful" patches involved solution leakage — the fix, or something close to it, was already stated in the issue report or the comment thread the model was given as context — and a further 31.08% passed only because the accompanying tests were too weak to catch an incorrect implementation. When the authors filtered those cases out, the measured resolution rate for that same model and scaffold fell from 12.47% to 3.97%, a drop of roughly two-thirds.

A second, more targeted study published in 2025 by You Wang, Michael Pradel and Zhongxin Liu — "Are 'Solved Issues' in SWE-bench Really Solved Correctly?" — built a differential testing tool, PatchDiff, that generates additional test inputs to check whether a model's patch actually behaves the same as the human-written reference fix, rather than just passing the fixed set of tests bundled with the task. Applying it to patches from three agent scaffolds on SWE-bench Verified, they estimate that 11.0% of patches marked as resolving an issue were in fact behaviorally incorrect, which inflates reported resolution rates by roughly 6.4 percentage points. Manual review of a 77-patch sample confirmed 28.6% were certainly wrong despite being scored as passes.

Both papers land on the same structural problem from different angles: the benchmark's pass/fail signal comes from unit tests scoped to the original human fix, and unit tests are, by design, an incomplete specification. A patch can satisfy the letter of the test suite — or exploit a leaked hint about what the human did — without doing what the issue actually asked for.

The benchmark's own steward pulled the plug

The clearest institutional signal came from OpenAI, historically one of the heaviest users of SWE-bench Verified in its own model announcements. In a post published on 23 February 2026, "Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities," the company said it would stop reporting scores on the benchmark. Its stated reasons track the independent audits closely: OpenAI's own review of a sample of previously "unsolvable" problems found the majority had test cases that were either too narrow (rejecting functionally correct submissions) or too broad (silently requiring implementation details never stated in the problem description) — the kind of underspecification that only someone who has already seen the reference solution could reliably satisfy. The post also reported that current frontier models could reproduce the original gold patch, or reconstruct precise problem-statement details, from the task ID alone with minimal prompting — evidence that the benchmark's public repositories and issue text have entered training data, not just that models are good at inferring intent.

By early 2026, multiple frontier models were clustering close together near the top of the leaderboard, according to independent tracking by Epoch AI — which is itself a symptom rather than a triumph: once most serious contenders bunch near the ceiling, the benchmark stops being able to distinguish between them, whatever the underlying cause of the compression.

What independent re-scaffolding shows

Epoch AI, a research organization that re-runs published benchmarks under its own harness rather than trusting self-reported numbers, evaluates SWE-bench Verified using a fixed Docker environment, a defined token budget and a deliberately excluded set of samples it judges too unreliable to score — infrastructure choices that, as the organization notes, vary across labs' own reported numbers even before contamination is considered. That is a useful reminder for reading any single-number benchmark claim: the same benchmark name can correspond to different scaffolds, different tool access and different token limits, so scores are only comparable when the harness is held constant, which vendor-reported numbers frequently do not disclose in full.

The response from the benchmark-building community has mostly been to build harder, less contaminated successors rather than patch SWE-bench Verified. Scale AI's SWE-Bench Pro, released in September 2025, sources its public task set from repositories under copyleft licenses specifically because that licensing makes inclusion in commercial training corpora less likely, and it supplements the public set with held-out and commercial task pools the model providers cannot have seen at all. The difficulty gap is stark: on SWE-Bench Pro's public set, Scale AI reported GPT-5 as the top performer at a 23.3% pass rate and Claude Opus 4.1 close behind at 23.1%, well below the scores the same class of model posts on SWE-bench Verified. That gap is itself evidence for the contamination argument — a benchmark that current models solve four-fifths of the time and a structurally similar one they solve roughly a quarter of the time are not measuring the same underlying difficulty, even though both are nominally "resolve a real GitHub issue."

A separate, harder-to-audit claim: task length over time

Alongside benchmark-specific pass rates, one other measurement has circulated widely enough to warrant its own scrutiny: METR's finding, published in "Measuring AI Ability to Complete Long Software Tasks" (Kwa et al., 2025), that the length of task — measured in the time a skilled human would need — that frontier models can complete with 50% reliability has been doubling roughly every seven months since 2019, with data from 2024–2025 suggesting a faster pace. This is a different kind of claim from a benchmark pass rate: it is a trend fitted to METR's own task suite (a mix of software engineering, cybersecurity and general reasoning tasks timed against human baselines), not a fixed leaderboard score, and METR is explicit that the fitted trend is sensitive to which tasks and which human timings are included, and that the exponential form itself is an empirical fit rather than a law. A subsequent analysis by Toby Ord, "Is There a Half-Life for the Success Rates of AI Agents?" (2025), proposes a specific mechanism for the pattern METR observed — a constant per-minute failure hazard, producing an exponentially decaying success rate with task length — while noting explicitly that whether the same model holds on other task suites is unknown. Both pieces of work concern trend extrapolation, not a validated benchmark score, and should be read with that distinction in mind: a doubling trend fitted to one research group's task set is a hypothesis about capability trajectories, not a certified measurement of what any single deployed system can do on a given day.

What can and cannot be said with the current evidence

• It is defensible to say that a substantial share of "resolved" SWE-bench and SWE-bench Verified issues — on the order of a quarter to a third, depending on the study and the model scaffold examined — do not reflect a correct, non-leaked fix, based on manual and automated audits by two independent research teams.
• It is defensible to say that SWE-bench Verified's usefulness for ranking current frontier models has declined, because the benchmark's own most prominent user stated so in writing and pointed to specific, checkable causes: narrow or overly broad test cases and reproducible gold-patch recall.
• It is defensible to say that a newer, less contaminated benchmark (SWE-Bench Pro) produces pass rates several times lower for the same models, which is consistent with — though does not on its own prove — a contamination explanation for the older benchmark's high scores.
• It is not yet defensible to treat a single reported percentage on SWE-bench Verified, from any lab, as a stable measure of "coding ability" without knowing the scaffold, token budget and evaluation date behind it, since those variables change the number independently of the underlying model.
• It is not defensible to extend the METR task-doubling trend into a firm prediction about when agents will handle week-long or month-long tasks unsupervised; the source paper presents that as an extrapolation with named sensitivities, not a forecast the authors endorse as reliable in itself.

None of this means AI coding agents are not improving — the gap between GPT-4-era resolution rates and current frontier scores on cleaner benchmarks such as SWE-Bench Pro is real and large by the numbers Scale AI has published. It means that the specific number most often quoted to demonstrate that improvement has, by the account of the people who built and later retired it, been measuring benchmark familiarity as much as software engineering skill for some part of its run. For anyone using these scores to make a claim — in a paper, a procurement decision or a comparison table — the working discipline the evidence supports is the same one researchers already apply to any other instrument: name the exact benchmark version, the harness, and the date, and where possible check whether an independent group such as Epoch AI has reproduced the number under its own conditions before treating it as settled.