OpenAI finds critical errors in popular SWE-Bench Pro benchmark

OpenAI analyzed SWE-Bench Pro, a popular test suite for evaluating the coding skills of AI models, and found critical problems: roughly 30% of the tasks contain errors. As a result, the company has withdrawn its support for and recommendation of the benchmark.

What is SWE-Bench Pro

SWE-Bench Pro (Software Engineering Benchmark Pro) is a standardized test suite designed to objectively measure how well AI models solve real-world programming tasks. Researchers, companies, and independent evaluators use it widely as one of the main tools for comparing the performance of different coding models.

The tasks in SWE-Bench Pro come from real GitHub issues. They require a model to do more than write code: it has to understand complex software systems, fix bugs, and handle non-trivial scenarios. Its closeness to real-world work and its objective scoring made the benchmark popular across the AI community, and it became the de facto standard for public model comparisons.

30% of tasks contain errors or defects

OpenAI's analysis found serious problems in roughly 30% of SWE-Bench Pro tasks. The errors range from incorrect test cases to poorly worded problems and flawed reference solutions. They make results on those tasks unreliable and give a misleading picture of what models can actually do.

Finding problems at this scale in a popular, authoritative benchmark is a significant blow to its reliability as an evaluation tool. If 30% of all tasks means roughly 90 or more faulty tests, that is a huge amount of compromised data. OpenAI previously endorsed SWE-Bench Pro and publicly recommended it as a reliable test for comparing models; now it is officially withdrawing that support. Results based on this benchmark, which are widely cited in research papers, blog posts, and comparison reviews, should therefore be interpreted with great caution.

Why this matters

Reliable, unbiased tests are the foundation of fair evaluation of AI model capabilities. When a benchmark contains systematic errors, results stop being comparable, and companies or researchers may draw the wrong conclusions about which model is actually better.

The incident shows that even popular, widely used tests need ongoing review, validation, and updates. Companies and researchers who relied on SWE-Bench Pro results when choosing models or evaluating their own systems should revisit their conclusions and be more careful when interpreting this data.

• About 30% of tasks in SWE-Bench Pro contain errors or defects
• OpenAI has withdrawn its official support for and recommendation of the benchmark
• Researchers and companies widely used the test to compare coding models

What it means

The problems found in SWE-Bench Pro point to a fundamental challenge in evaluating AI capabilities: building truly representative, unbiased, and error-free tests is extremely hard and takes enormous effort. The community needs to work closely together on more reliable evaluation methods that reflect what models can really do, without artifacts and errors in the tests themselves.

This may lead to a review and reassessment of comparative results for various AI models previously published on the basis of SWE-Bench Pro. Developing improved benchmarks for evaluating the coding skills of AI systems, and independently verifying them, will take significant additional effort.