CRM-Agent-Bench tests 16 AI agents: the best strictly passes 83.6% of tasks

An agent in a chat can sound flawless and still leave a corrupted record in the CRM, the wrong order status or an unnecessary refund. The new CRM-Agent-Bench benchmark measures exactly that gap between "said" and "did." Its authors, Monojit Banerjee, Zhujun Lan and Defu Cao, assembled 555 stateful, multi-turn tasks across nine business domains. Sixteen evaluated systems scored between 64.32% and 83.60% strict success. The code is published in the salesforce-misc/crm-agent-bench repository, and the project has a dedicated page.

A contract instead of "looks right"

Most agent benchmarks look at either the final answer or the final state of the environment. The authors consider that insufficient for production deployment and propose evaluating four dimensions at once:

• outcome, meaning the result the agent arrived at;
• trajectory, meaning the steps it took to get there;
• operational viability: latency and cost;
• compliance with the policies specified in the task.

CRM is almost ideal for this kind of test: records get modified, operational rules sit on top of them, and mistakes have financial consequences. That is why each of the 555 tasks comes with its own executable contract. The contract is built from checks of six types:

• actions the agent must perform;
• arguments of tool calls;
• order of operations;
• content of the response to the user;
• target state of the data after the work is done;
• protected state that must not be touched.

The last item matters more than the rest. An agent that closed a ticket and overwrote someone else's record in the process technically did what was asked, but in a real company it caused an incident. Success in the benchmark is defined as a strict conjunction: a task counts only if every check declared for that specific task passes. No partial credit is awarded for "almost right."

Sixteen systems, a ceiling of 83.6%

The benchmark evaluates not a bare model but the whole system: the agent harness, the tools and the dialogue logic. That is closer to what companies actually buy and deploy. Besides the share of successful tasks, the benchmark records evaluation coverage, latency and the cost of a run.

Results on the verified final release are as follows: sixteen systems fell within a range of 64.32% to 83.60% strict success. A spread of almost twenty percentage points shows that the tasks are not saturated and that the benchmark tells systems apart. The leaders are tightly grouped, but they diverge sharply on price and on reliability in individual domains. One system can be strong in one subject area and weak in another, and the average percentage hides this.

The tasks are labeled with 36 capability tags, so failures can be analyzed by specific skill and domain rather than by a single overall number. The authors call the framework auditable and portable: its structure can be extended as models hit the ceiling.

How to use this when choosing an agent

For teams deploying agents in customer support, sales or back-office work, the main lesson is that answer accuracy does not guarantee that a transaction is correct. What needs to be evaluated is the final state of the data, the order of actions and the absence of side effects.

The second practical takeaway: look at the numbers as a pair, success plus cost. If the leaders differ substantially in price but barely in quality, choosing the most expensive option is poorly justified.

Third: the overall percentage does not replace a per-domain breakdown. Before trusting an agent with payments, refunds or changes to customer records, it is worth running tasks that match your own profile and checking the protected state. The contract structure from CRM-Agent-Bench can be carried over to your own processes: for each scenario, describe the expected actions, the permissible arguments and the data that must not be touched.

A limitation remains: a maximum of 83.6% means that roughly one task in six fails under the strict criteria even for the best system. For processes where an error costs money, that is not enough without human oversight and rollbacks.