AdaPRA: A Supervisor Agent Boosts AI Performance on Data Science Tasks by 24.5 Percentile Points

Agents, Quiet Failures

Agents tasked with handling an entire data science pipeline — from loading the data to submitting predictions on Kaggle — routinely lose points not because they don't understand the algorithms, but because of process failures: stopping at the first model that works instead of trying something stronger, mixing up training and validation splits, or letting data leak into cross-validation. Worse, many of these mistakes leave no trace in the execution log — the pipeline runs cleanly and spits out a prediction file, and the fact that the predictions are meaningless only becomes apparent on manual inspection. Researchers from Carnegie Mellon University cite a telling example: an agent tasked with classifying voice commands across 12 categories loaded all 158,538 test audio recordings as silence because of an incorrect file path.

The model classified every single one as "unknown command," while the log looked like everything had gone exactly as planned.

How the Verifier Agent Works

The solution is called AdaPRA (Process Reward Agent with Adaptive Rubrics). It's neither a static checklist nor an ordinary LLM judge that simply reads a text report of what the agent did. AdaPRA is itself an agent with terminal access: it opens the primary agent's working directory, runs its own code to recompute metrics (for instance, re-scoring cross-validation quality from held-out predictions), and only then renders a verdict on each rubric item. This matters because an agent's self-reported account of its own work can't be taken at face value — the agent can be genuinely wrong about what it actually did.

Verification doesn't run continuously; it happens at fixed intervals — every few steps of the main agent, plus a final check before submission. To avoid overwhelming the agent with noise, the report only includes rubric items that are actually relevant to the current task and that the verifier judges likely to have been violated — with a violation probability above 40% and significance at least half of the maximum.

Where the

Rules Come From, and Why They Transfer Across Tasks

The researchers didn't write the rubrics by hand — that would have been unrealistic across hundreds of task types. Instead, they ran the baseline agent many times on 38 easy-level tasks (the DSPredict-Easy set, mostly tabular Kaggle Playground Series competitions) and compared the best attempts against the worst by final leaderboard rank. An LLM judge (Claude Sonnet 4.

6) extracted 342 "good" patterns and 564 problems from 906 observations of successes and failures, then grouped them into 17 concrete rules plus 2 general rules about using prior attempts — 19 base rubrics in total. One notable detail: almost all of the identified problems were visible only in the code and execution logs themselves (344 of 564), while the successful approaches showed up in the working directory's artifacts (235 of 342). In other words, looking only at the dialogue with the agent, or only at the files, would miss most of the signal.

These 19 rubrics were transferred unchanged to DSPredict-Hard — a set of 54 far more complex tasks (vision, text, audio, molecular data) — and they worked. AdaPRA can also add rules on the fly: on the House Prices task, for example, the verifier itself added a new rule called "insufficient prediction variance" — a check for whether the model is collapsing all its price predictions toward the mean instead of producing a realistic spread.

Results and Limitations

Across five attempts per task, AdaPRA with the baseline Qwen3-Coder-Next model (80 billion parameters) raised the average Kaggle leaderboard percentile by 24.5 points on easy tasks and 14.8 points on hard tasks — matching or beating GPT-5.1 running in enhanced reasoning mode. The system also outperformed DataPRM, a competing 4-billion-parameter verifier model that scores agent steps on a 0–1 scale without flexible rubrics. A separate experiment showed that rubrics built on one model transfer successfully to another — DeepSeek-V3.2.

For engineering teams building autonomous data analysis agents, the practical takeaway is clear: checking the dialogue log alone catches far from all the mistakes — you need a supervisor with the ability to actually execute code and inspect the files in the working directory. The second takeaway is that preserving context between attempts — rather than starting from a blank slate every time — delivers a meaningful boost all on its own, even before adding any external quality control.