AgentProofBench: CMU researchers test whether web agent verifiers can be reused

Researchers from Carnegie Mellon University, led by Jingjie Ning, have introduced AgentProofBench, a benchmark for web agent verifiers. It answers a practical question: can you write an acceptance rule once and then reuse it as requirements change and as agents run in real execution? The resource covers 97 tasks, 79 task families, and four web services. The verifier programs were generated by DeepSeek Flash and DeepSeek Pro.

The idea behind the benchmark is that judging a verifier on verdict accuracy alone isn't enough. An agent operates in a chain: it receives an instruction, acts on a website, passes a check, and only then is the result delivered to the user. That's why AgentProofBench pairs tests of a "frozen" acceptance rule with agent runs launched from the same saved initial states. Every test failure can be traced back to its consequences in execution.

Reuse or rebuild

The first dimension concerns resilience to changing requirements. Across 48 task families, two tests were applied: complete-panel and decision-replay. They show whether the original verifier program remains usable when a requirement changes, or whether it has to be rewritten from scratch.

• Flash: 82 of 96 original programs survived (about 85%).
• Pro: 79 of 96 survived (about 82%).
• The gap between reuse and rebuild on the BA metric for Flash ranges from -5.47 to 1.57 percentage points.
• For Pro, the same range is -10.16 to 0.00 pp.

Flash's interval includes zero, meaning there's no significant loss from reuse in this data. For Pro, the upper bound sits at zero while the lower bound drops nearly 10 points. In the worst case, reuse costs noticeably more, and the benchmark makes that cost measurable.

Panel errors as a delivery indicator

The second dimension concerns execution and is built on 22 task families. The authors looked at program assignments where both saved initial states deliver correctly, and searched for what distinguishes those cases from the rest. The distinguishing factor is panel correctness in the deployment state.

• Of 33 assignments with an error-free panel, 32 delivered correctly in both states.
• Of 11 assignments where the panel had errors, this happened in only one.
• All program-gate regressions relative to reference rules, and all cases of withheld delivery, fell into the group with panel errors.

A 97% versus 9% split on a small sample of 44 assignments can't be considered conclusive. But it shows that checking a verifier against a test panel before deployment gives a practical signal of whether the program is safe to put into production.

A baseline for gates

The authors compared feedback strategies for the agent. Among the tested universal options, the highest share of correct delivery came from neutral continuation, where the agent simply keeps working without gate intervention. This sets a floor: any verifier-gate placed before delivery must beat it, or it only adds complexity and the risk of false rejections. Until now, no such baseline existed for web agents, and teams often judged gates by eye.

What's in the release and how to use it

Alongside the benchmark, the team published instructions, environment states, verifier programs, reference rules, and agent trajectories. Code and the project page are available on GitHub and at ethanning.github.io/AgentProofBench. The preprint is registered as arXiv 2610.34634 and appeared in the Hugging Face Daily Papers digest.

For teams building web agents with automated result verification, the benchmark offers three concrete steps. First: measure how many generated verifiers survive a change in requirements, and don't rebuild them unnecessarily. Second: check the test panel for errors before deployment, since these errors coincided with every regression and every withheld delivery. Third: always compare a gate against the "no intervention" option before assuming it improves results. The limitations are also clear: four web services and two models from the same family give a narrow picture. But as a methodology for evaluating requirements and execution policy, the work sets a clear frame.