OpenAI drops the main coding benchmark, and it changes the rules of the game

When a company whose models consistently sat at the top of a leaderboard publicly walks away from it, that is not just a corporate decision. It signals a systemic problem. OpenAI has announced that it is ending evaluation of its models on SWE-bench Verified, the benchmark that for the past two years served as the main yardstick for how well AI can write and fix real code. The reason is simple and worrying at once: the benchmark no longer measures what it is supposed to measure.

SWE-bench began as an ambitious attempt to move beyond synthetic tests. Instead of asking a model to solve an abstract LeetCode puzzle, it offered real bug reports from popular open-source Python projects such as Django, scikit-learn and sympy. The model had to understand the bug description, find the right file in the repository and write a patch that passes the tests. The Verified version came later as a cleaned-up variant, with tasks checked manually by humans. Labs competed on it, proudly publishing the percentage of solved tasks in every press release.

But behind the facade of impressive numbers, problems were piling up. OpenAI's internal analysis found two critical weaknesses. The first is training data contamination.

SWE-bench Verified tasks are based on public pull requests in open repositories. That data inevitably ends up in the training corpora of large language models. Put simply, models may have seen the correct answers before they were ever tested.

This is the classic data leakage problem, but with SWE-bench it has reached a scale that makes the results statistically meaningless. The second problem is the quality of the tests themselves. Some tasks had incorrect or incomplete tests that could let wrong solutions pass or reject correct ones.

Once a benchmark becomes popular enough, people start optimizing for it, and not always by honest means.

The context of this decision matters. AI coding is in a period of explosive growth. Dozens of startups, from Cognition with its Devin to Poolside and Magic, are raising hundreds of millions of dollars, and almost all of them use SWE-bench results as the main argument in their pitch decks. Major labs, including Anthropic, Google DeepMind and OpenAI itself, publish results on this benchmark with every new model launch. SWE-bench Verified effectively became the currency of trust in AI programming. Now OpenAI is saying that currency has lost its value.

The company recommends moving to SWE-bench Pro, an updated version of the benchmark that, according to its creators, fixes both problems. The new tasks are selected specifically to minimize overlap with public training data, and the tests go through stricter verification. But a natural question arises: how long will SWE-bench Pro stay clean? The history of machine learning benchmarks is a history of gradual decay. ImageNet, GLUE, SuperGLUE, MMLU: each eventually stopped telling truly strong models apart from those that were simply well trained on that particular test.

For the industry, the consequences reach well beyond a single benchmark. Investors putting money into AI coding startups now have to ask what actually stands behind the impressive numbers they were shown. Companies integrating AI assistants into their development workflows are forced to revisit their selection criteria. And researchers get another reminder that in the race for benchmark leadership, the link to real usefulness gets lost.

There is a deeper question too. If the world's leading AI lab admits that the standard tool for measuring progress is broken, how are we supposed to know whether models are getting better at all? In a world where a new "revolutionary" model with record scores arrives every quarter, the lack of a reliable ruler is not a technical detail but a fundamental problem.

OpenAI deserves respect for the honesty of this admission. But the very fact that the industry relied on a contaminated benchmark for so long points to a systemic shortage of critical thinking in the community. The move to SWE-bench Pro is a step in the right direction.

But real progress will begin when we stop reducing AI evaluation to a single number on a single test and start building multidimensional, manipulation-resistant evaluation systems that reflect how well models actually help developers in their daily work.