Cognition releases FrontierCode 2.0: the best score is just 45.5%

Cognition, the company behind the Devin agent, has published FrontierCode 2.0, the second generation of a benchmark that evaluates not whether tests pass but the "mergeability" of code: would the maintainer of a real project accept this pull request? Claude Opus 5.5 posted the best result at 45.5%, down from 54.6% in the previous version, FrontierCode 1.1. The gap between models is now more visible, and the ceiling is far away: the benchmark is not saturated.

Why Cognition raised the bar

In Devin's experience, agentic software development is far from a solved problem. Models are poor at guessing user intent, violate non-obvious constraints and fail to foresee the long-term consequences of their edits, so a human has to step in. Most coding benchmarks don't capture this: they provide a detailed specification and saturate quickly. One example is the vf2-speedup-networkx task from Terminal-Bench 4.0, which fully describes the approach, the format and the verification conditions, including a speedup of at least 5000x.

FrontierCode is built differently. More than 20 open-source maintainers spent 40+ hours on each task, assembling it from their own repositories. In version 2.0, tasks come from 55% more repositories and include security patches and performance optimizations. There are 43% more blocking criteria. On average, the reference solutions change 67% more files and 89% more lines, and the codebases are 25% larger. Cognition also strengthened the audit: in 80% of tasks the reference solution differs from the upstream PR (60% in 1.0). One of the automated audits even found a vulnerability in an upstream implementation, and the maintainers were notified.

Table: who lost how much

Scores below are in the format "FrontierCode 1.1 → FrontierCode 2.0":

• Claude Opus 5.5 — 54.6% → 45.5%. The leader, on the Pareto frontier at the expensive end.
• GPT-6 Astra — 53.3% → 42.8%.
• Grok 4.7 — 47.6% → 28.6%.
• Gemini 4 Argon — 47.5% → 26.3%.
• Muse Wonder 1.0 — 47.0% → 24.3%.
• SWE-2 — 50.0% → 23.9%.
• DeepSeek V4.1 Flash — 43.9% → 19.3%.
• Kimi K3 — 44.2% → 17.7%.

On the cost-versus-quality Pareto frontier, Opus 5.5 is joined by Luna, Haiku and Sol at the cheap end. Cognition singles out GPT-6.1 Sol for its strong result relative to cost. The takeaway on open models is blunt: there is a lot of room to improve, especially for Western OSS models.

Thoroughness over obedience

The main change is the growing share of "thoroughness" criteria: the number of blockers of this type rose from 168 to 501, a threefold increase. They check what the agent should infer from the instructions and the code even though it isn't stated directly. For example, fix a bug everywhere it occurs, not just in the place named, or support neighboring input types if similar code already handles them. The share of failed checks here rose from 11.7% to 19.2%, the highest failure rate of any category. For core behavior it is 14.9% versus 9.5%, and for backward compatibility 5.9% versus 2.9%.

The other blocker categories changed as follows: core behavior 399 → 353, regression avoidance 174 → 259, test quality 202 → 213, lint/format/typecheck 112 → 183, scope 56 → 123, backward compatibility 85 → 76, compilation 86 → 69, test integrity 48 → 48.

More effort isn't always better

FrontierCode does not reward the sheer volume of code written, so the score can fall as reasoning effort rises. The cause is the scope criteria. For Claude Opus 5, the share of scope criteria passed falls from 69.2% at low to 51.1% at max; for Opus 4.8, from 81.9% to 70.9%; for Claude Fable 5, from 79.1% to 63.4%; and for GPT-5.6 Sol, from 79.7% to 68.3%. Implementation criteria, meanwhile, go up: for Opus 5, from 79.3% to 83.0%. With more "effort," models touch more files, sometimes unnecessarily. Among the less significant causes, Cognition cites more frequent cheating (accidentally loading the commit with the solution) and "overthinking," where the model talks itself into a wrong reading of the task.

In version 2.0, the scope requirements were relaxed so as not to penalize unexpected but acceptable behavior: the average failure rate fell from 9.0% to 3.8%. On a pool of Claude Opus 5.5, Claude Fable 5.1, GPT-6.1 Sol and GPT-6 Astra, thoroughness fails in 26.0% of checks at low and in 15.3% at max.

The Mock Service Worker task is a telling example: the goal was to fix a bug where resetHandlers() does not restart generator resolvers. At every effort level from low to max, Opus 5.5 did not pass a single one of its five runs. The fix worked as intended, but the grader flagged the approach to the change itself.

What teams should take away

If you are choosing a model for autonomous coding, don't look at pass@1 on clearly specified tasks; look at behavior with an incomplete prompt. In practice: build in code review for side-effect edits and missed "neighboring" cases, don't default to maximum reasoning effort without measuring it on your own repositories, and compare models by cost per run, not just by score. For an open-source stack, the 2.0 results are a reason to budget extra time for manual finishing.