MLE-World predicts the outcome of an ML experiment without running it: 77.9% accuracy

Agents that write and improve code for machine learning tasks on their own hit an expensive wall: every candidate has to be actually trained and evaluated. The MLE-World team proposes replacing some of those runs with a prediction. Given a program's code and a description of the data, their world model produces what the terminal would print: the error trace, training logs and validation score. On held-out programs, it matched the real output 77.9% of the time. Large models given the same task via a prompt scored 41.1%.

One call instead of a training run

The bottleneck for ML agents is the feedback loop. At each step an agent can check only a few options, because each one is verified by training a model, which costs minutes to hours of GPU time. As a result, the agent either considers few candidates or spends its budget on computation that mostly ends in "it didn't work."

MLE-World changes the arithmetic. A prediction costs one inference call, not a training run. That makes test-time scaling, meaning generating several variants at each step and picking the best, practical for research agents. Previously, this technique was held back by the cost of verifying every candidate.

The model does not predict an abstract "quality score." It predicts the program's text output. That includes failed runs: an import error, a shape mismatch, running out of memory. It is useful for an agent to know in advance that a candidate will break, so it can skip the real run.

Checking how far to trust a prediction

Every world model has a weak spot: it can be confidently wrong. So the system includes a second component that estimates how likely a prediction is to be correct. It is not there for show. Real execution of a candidate is done selectively, guided by this confidence estimate: when trust in the prediction is low, the agent pays for actual execution.

On top of this pairing, the authors built five test-time scaling policies. All of them compare several candidates at each step by their predicted outputs rather than their real ones. One of them adds an exploration bonus to the selection, which is also built on the confidence estimate. It guards against an agent getting stuck on variants the model is simply sure about.

The numbers and what they mean

• Match between the prediction and the real terminal output on held-out programs: 77.9%.
• Baseline, much larger prompted models: 41.1%.
• Gain in the agent's observed average result with MLE-World compared with an agent without a world model: up to 22.1%.
• The amount of real execution is lower than for the agent without the model.
• Number of test-time scaling policies in the paper: five.
• Preprint: arXiv 2610.37175, project page mle-world.github.io.

The key detail is the comparison with large models. A specialized predictor trained on "code and data → output" pairs is nearly twice as accurate as general-purpose LLMs, which have not seen enough real runs. Knowledge of how pipelines behave is absorbed better from execution logs than from general pretraining.

The phrase "up to 22.1%" describes the best case, not a guarantee on every task. Even so, the direction fits the economics well: cheap inference replaces expensive GPU hours.

If you build agents for AutoML or coding, this approach has a direct application. It is worth collecting "code, data description and run output" triples from your own logs, because that is exactly the dataset that trains a world model. In the agent loop, separate cheap filtering by predictions from expensive verification by a real run, and set the trust threshold with the confidence scorer. That way the GPU budget goes to candidates where the prediction is ambiguous or the choice really matters, and obviously broken variants are weeded out before any run.