LiveMACEBench Tests Five LLM Agents on a Live Market for 30 Days

Researchers have introduced LiveMACEBench, a benchmark that evaluates LLM agents not by their final result but by how they make decisions. Live financial markets serve as the testbed. Five frontier models worked as persistent agents for 30 days in a row, while the authors saved full decision traces and broke them down by mechanism. The main result: realized returns often diverge from measured capabilities, and similar outcomes can come from very different behavior patterns.

Why a market instead of a static set of tasks

Classic agent benchmarks run in a closed environment with fixed conditions. The agent solves a task, gets a score, and that is the end of it. In an evolving environment, the picture is different. The outcome there emerges from a feedback loop: the agent's behavior affects the state of affairs, while external conditions change independently of it. If you look only at the final number, you cannot tell what drove it: skilled use of tools, effective memory of past steps, rule compliance, or simply luck on a particular stretch of the market.

A live financial market is a natural fit for this kind of test. It changes constantly, it cannot be memorized in advance, and the data cannot be tailored to a model during the test. The authors call it a "naturally evolving proving ground" for persistent agents.

How LiveMACEBench works

The experiment involves five frontier LLMs. They run on continuous trajectories rather than in separate episodes. Each model goes through comparable configurations in which individual capabilities are switched on or off.

• Tool Use: use of external tools.
• Persistent Memory: storing and applying memory across steps.
• Rule Following: compliance with given constraints and rules.
• Multi-Agent Collaboration: interaction among several agents.
• Duration: 30 days of live evaluation.
• Metrics: realized outcome plus diagnostics for specific mechanisms, based on full decision traces.

The second group of metrics is the main one here. The authors look not only at what the agent earned, but also at how it handled tools, what it recalled from memory, whether it broke the rules, and how it coordinated its actions with other agents.

The gap between outcome and capability

The key finding is called the outcome-capability gap. Returns often did not match the assessment of specific capabilities. The reverse also happened: similar final results concealed noticeably different patterns of mechanism use. Two models could arrive at a similar number, one through careful work with tools and memory, the other through something quite different.

Decision tracing also revealed different bottlenecks. Each capability had its own, so a single overall score of "agent quality" does not reflect the real picture. The authors separate three things that are often conflated: access to a mechanism, effective use of that mechanism, and final performance. By their conclusions, these quantities are not interchangeable.

The abstract does not give model names or numerical return values, so it cannot be used to compare vendors. The value of the work lies in its methodology: it turns a live market from a leaderboard into a diagnostic environment.

What teams building agents should do

The practical takeaway is simple: log not just the result but the entire path to it. Whether your agent trades, writes code, handles support, or manages processes, full decision traces are just as necessary for debugging as the success metric.

Second: test mechanisms separately. Having memory or a set of tools in the architecture does not guarantee that the agent uses them sensibly. It helps to set up metrics such as "how often and how appropriately a tool is called" and "whether retrieved memory influences the decision."

Third: do not trust a single run in a changing environment. Strong returns over a short period may be noise, while weak returns may hide strong capabilities that simply did not line up with the market phase. Trace-based diagnostics help tell the two apart and show what exactly to improve: memory, rule discipline, or coordination between agents.

For those choosing a model for an agent product, the conclusion is just as practical: compare candidates on your own traces, not on a single public leaderboard.