RoboJudge-9B Judges Robot Videos More Accurately Than Gemini: 0.719 Correlation With Humans

Video generation models are being positioned as the foundation for robot "world models." The neural network first renders how a task will play out, and that footage can then be used to plan actions or train policies. The idea works only under two conditions: the clip must be physically correct, and the actions in it must match the instruction.

Checking this by hand at scale is impossible, so the judge role is increasingly handed to multimodal LLMs. Researchers at the Chinese University of Hong Kong set out to find out how closely such judges actually agree with humans, and ended up building an entire toolkit called RoboJudge. The best model in the set, RoboJudge-9B, reaches a Pearson correlation of 0.

719 with human ratings.

Where general-purpose judges go wrong

The core of the toolkit is RoboJudge-Bench. It contains 800 human-annotated videos drawn from 8 source corpora and generated by 10 different video models. The clips cover three types of "body": simple grippers, human hands and dexterous multi-fingered hands. Each clip is scored on two axes: Physical Adherence and Instruction Alignment.

The results were uneven. General-purpose MLLMs agree with humans noticeably less on physics: an object may pass through fingers or change shape, and the judge model fails to notice. Specialized video judges trained on general physical reasoning transfer to robotic scenes unreliably. In other words, a ready-made "physics" judge built on ordinary video does not guarantee that it will understand how a manipulator contacts an object.

How a benchmark became a 9-billion-parameter judge

In response, the authors built a training dataset, RoboJudge-Train: 12 thousand videos and 25 thousand human-verified ratings. The ratings come with model-assisted rationales, so the judge learns to explain its verdict instead of just outputting a number. The team fine-tuned a 9-billion-parameter MLLM on this data in two stages, supervised learning first and then reinforcement learning. The result is RoboJudge-9B.

The headline number: among all judges tested on RoboJudge-Bench, it has the highest overall agreement with humans. Its Pearson correlation is 0.719. That is 0.089 higher than Gemini-3.7-Flash, the strongest proprietary baseline in this comparison. The authors stress that the gain is statistically significant. These figures put Gemini-3.7-Flash at a correlation of about 0.63. A small specialized model beat a closed competitor on its home turf.

Training on RoboJudge-Train also helped on external tests, including VideoPhy2 and Physion-Eval. So the judge did not simply memorize the quirks of a single benchmark.

What the toolkit includes

• RoboJudge-Bench — a benchmark for evaluating MLLMs as judges of embodied video: 800 human-annotated clips, 8 source corpora, 10 video generators, three manipulator types (grippers, human hands, dexterous hands).
• Physical Adherence — the first scoring axis: how physically plausible the clip is.
• Instruction Alignment — the second axis: whether the action in the clip carries out the given instruction.
• RoboJudge-Train — the training dataset: 12K videos, 25K human-verified ratings and model-generated rationales.
• RoboJudge-9B — a 9B-parameter MLLM trained with SFT and RL; Pearson 0.719 with human ratings, +0.089 over Gemini-3.7-Flash.
• VideoPhy2 and Physion-Eval — external benchmarks where training on RoboJudge-Train also produced gains.

The authors have already used RoboJudge-9B as an evaluator to rank video generators on the two axes. The code is on GitHub under the Sphere-AI-Lab organization, and there is a project page at spherelab.ai/RoboJudge. The work was done by Siyuan Ma, Yu Qi, Cheng Ye, Ruotian Peng, Xinyi Xu, Zitiantao Lin, Zedong Cai, Yijian Huang, Weiyang Liu, Siyuan Wang and Yandong Wen.

Who a robot-video judge is useful for

Teams that train video models as simulators for robots get a cheap way to filter out bad clips and compare checkpoints without an army of annotators. A 9B-parameter model can be run in-house, instead of paying for a closed model's API on every evaluation. The result is also useful for RL: a reliable automatic judge can serve as a reward signal when fine-tuning generators. Caution is still warranted. A correlation of 0.719 is clearly higher than the competitors', but it is not agreement with a human. For critical tasks, such as training policies on synthetic data for real manipulators, spot checks by humans remain mandatory.