NUS Teaches a Model to Find What It Described: gIoU Rises From 57.5% to 68.2%

Researchers at the National University of Singapore have introduced Generalized CycleGRPO, a reinforcement learning method in which a multimodal language model learns to describe a selected region and to find it from a text description at the same time. The method covers spatial, temporal, and spatio-temporal settings. On GroundingSuite, it raises the overall gIoU of the SAMTok model from 57.5% to 68.2%, and on GRES the average gIoU rose from 71.2% to 78.7%. The authors are Xin Zhang, Haochen Wang, Beibei Lin, Zhaoxiang Zhang, Robby T. Tan, Jason Li, and Dacheng Tao. The paper extends CycleGRPO, which was accepted at ECCV26.

A Description That Has to Lead Back to the Region

Fine-grained multimodal understanding requires two skills. The first is explaining what is in a given area of a frame or video. The second is finding that area from a phrase. These two directions are usually trained separately, each with its own loss functions and its own labeled data.

Generalized CycleGRPO closes them into a loop. The model is given a region and generates several candidate descriptions. Each description is then fed back in as input, and the model tries to recover the original region. The match between the recovered area and the original yields a verifiable reward for caption quality. Separate localization rewards improve the reverse mapping, from text to region.

This setup has no reference captions, and it needs no external language model as a judge. The task itself checks the description: if the original region cannot be found from the text, the description is bad.

From Image Masks to Video Intervals

The main engineering move is a shared region space. For images, a region is a mask or a bounding box. For video, it is a time interval plus spatial evidence with timestamps. The same loop works across all three domains: spatial, temporal, and spatio-temporal.

In video, the loop has an additional effect. If an event repeats, for example a person opens the same door several times, a vague caption fits every repetition, and localization from it will miss. The reward pushes the model to write descriptions that distinguish the target episode from similar ones.

To test this property, the authors built DOVE (Distinctive Occurrence-level Video-caption Evaluation), a contrastive benchmark verified by humans. It measures how unambiguously a caption points to a specific occurrence of an event.

Numbers Across All Benchmarks

• GroundingSuite: the overall gIoU of SAMTok rose from 57.5% to 68.2% (+10.7 pp).
• GRES: average gIoU rose from 71.2% to 78.7% (+7.5 pp).
• Charades: 51.9% tIoU versus 45.6% for the Qwen3-VL initialization.
• DOVE: 33.3% Temporal Evidence Coverage versus 14.8% for Qwen3-VL, more than double.

On temporal tasks, Qwen3-VL serves as the base; on spatial tasks, the comparison is against SAMTok. The sharpest jump is on DOVE, which tests exactly the ability to tell repeating events apart.

Where This Could Be Useful

The method's main value is that it removes the dependence on expensive annotation. Text-based segmentation, phrase-to-object grounding, and moment retrieval in video usually require "region–description" pairs, which are labeled by hand or generated by a strong teacher model. The loop derives its training signal from the structure of the task itself.

In practice, this is of interest to teams building video analytics, search over camera recordings, and editing tools. These products need to find the exact clip and object from a phrase like "the second time the courier leaves a package." Tying language to a specific region matters just as much for robotics and for agents that act in interfaces and physical environments.

The authors have published the models, code, and the DOVE benchmark on the project page, so the results can be checked on your own data. They propose the idea of region–text cycle consistency as a general principle of self-evaluation for multimodal models.