Agent4Sci: Claude Opus Beats Records on 20 Scientific Benchmarks by 10.4%

Frontier coding agents are increasingly pitched as automated researchers that can run the full cycle. The authors of Agent4Sci set out to test that claim on tasks where engineering diligence alone is not enough: the agent has to improve a published best scientific model. The testbed covers 20 benchmarks from 11 scientific fields. The result is mixed. Agents improved the record on every benchmark, and Claude Opus reached an average relative gain of 10.4%, but almost all of the gains came from standard ML engineering techniques, not research insight.

A test without a leaderboard

Agent4Sci deliberately does not build a leaderboard. The authors ask a different question: how much progress can agents actually make in conditions close to real work ("in the wild"), and where do they hit a ceiling? These are long-running tasks in which the agent decides for itself what to try, how to allocate compute, and when to stop. The comparison baseline is the published best models, so the bar is set by human researchers, not by a weak baseline.

The authors also note that no run was flagged as academic misconduct. For agentic benchmarks, this matters: the authors found no grounds to suspect test-specific overfitting or dishonest tricks.

A bag of engineering tricks instead of ideas

The fact that agents win everywhere sounds impressive, but trajectory analysis shifts the emphasis. The gains are almost entirely explained by a set of techniques familiar to any ML engineer. Ideas grounded in domain knowledge, whether physics, biology, or materials science, barely appear among the improvements.

The authors identify three recurring limitations:

• Miscalibrated novelty. The agent judges poorly what is genuinely new and what is already standard practice.
• Underuse of resources. The available compute and time budget is not fully spent.
• Rapid collapse into local engineering search. The agent slides into a series of small tweaks instead of looking for substantive hypotheses.

Plugins that doubled the gain

The most practical part of the paper is an experiment with lightweight plugin skills. There are three: research idea generation, experiment management, and scheduled checklists. Without retraining the models or rebuilding the architecture, these add-ons lifted Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%. Beyond the metric gain, the character of the agents' interventions changed: they became more varied and closer to the specifics of the given field.

The takeaway: a significant part of the gap between engineering and research is closed not by a new model, but by scaffolding that forces the agent to plan, log its experiments, and return to a checklist. At the same time, the absolute numbers remain modest, and Codex with the skills still trails Claude Opus without them.

What to take away

Teams building agents for scientific and R&D tasks get a concrete guideline: measure not only the final metric, but also the type of improvements found. If the gain comes from hyperparameter search and standard techniques, that is valuable, but it is not a scientific discovery.

The second conclusion concerns the architecture of agent systems. Experiment management layers, mandatory hypothesis-generation steps, and periodic checklist reviews cheaply improve returns and apply to existing models. They are worth testing before waiting for the next generation of LLMs.

Finally, the testbed itself is open. The code is available on GitHub (maszhongming/agent4sci), and the data is on Hugging Face (MingZhong/agent4sci). It gives labs a ready-made way to test their own agent on 20 tasks from 11 fields and see where it acts like an engineer and where it starts proposing ideas typical of a researcher.