Moonshot Releases WorldVQA: 3500 Questions About the World That Trip Up Models

Kimi Team at Moonshot AI has released WorldVQA, a benchmark that tests how much visual knowledge about the world multimodal language models have. The set contains 3500 image-question pairs, each with a single verifiable answer. The test asks a simple thing: does the model recognize the specific object in the picture, or does it fill in an answer from familiar visual patterns? According to the authors, on rare, "long-tail" knowledge even frontier models often fail to reach 50% accuracy. The dataset and evaluation scripts are published, along with a paper on arXiv.

What's inside the dataset

The questions are built to test encyclopedic breadth, not reasoning about an image. The authors excluded subjective questions and ambiguous scenes. All images and question-answer pairs went through multi-stage manual review.

The examples show the range: identifying a bird species (Chestnut Shortwing), a flower (freesia), a natural landmark (the Cape of Good Hope), the ballet Swan Lake, a video game character, the anime Your Name, and the EgyptAir logo. Some prompts demand maximum precision, such as naming a consumer gadget down to the exact model, up to the iPhone 17 Pro. Some questions are written in Chinese: about an ancient Buddhist temple, a museum exhibit from the Warring States period, the J-20 fighter jet, and a stadium in Shanghai.

Key parameters of the set:

• 3500 image-question pairs;
• 2240 questions in English (64%) and 1260 in Chinese (36%);
• 9 categories, from nature and architecture to brands, sports, and public figures;
• difficulty: 31.16% easy, 40.77% medium, and 28.07% hard questions;
• a split into Head (well-known) and Tail (rare) knowledge, to show how quality drops as an object becomes less popular.

The largest categories are "Locations and architecture" (14.63%), "Entertainment and media" (14.60%), "Culture and art" (14.46%), and "Public figures" (14.29%). The smallest is sports, at 4.06%.

The problem is not only accuracy, but also self-assessment

The second part of the study covers calibration: how closely a model's stated confidence matches its actual share of correct answers. It uses two metrics. ECE (Expected Calibration Error) shows the average gap between confidence and accuracy; the ideal value is 0. Slope (weighted average slope) shows how closely accuracy tracks confidence; the ideal value is 1.0.

All the models tested show the same result: systematic overconfidence. Most predictions cluster in the 90–100% range of stated confidence, even though actual accuracy is noticeably lower. The authors' own model, Kimi-K2.5, has the best scores on both metrics: ECE 37.9% and Slope 0.550. That is the best result in the comparison, but it is far from ideal: with zero calibration error, ECE would be close to zero and Slope close to one. The authors themselves acknowledge that models still have a long way to go before they are "honest" and aligned with the limits of their own knowledge.

Practical takeaways

For teams building multimodal models into products, the main lesson concerns how far to trust the answers. When a model names a car model, a plant species, a logo, or a work of art, high confidence in the answer says little about whether it is correct. This matters most for rare objects: that is where accuracy drops while confidence stays high.

What to do in practice:

• run your candidate models on WorldVQA if your product relies on recognizing specific entities, and look at Head and Tail separately;
• do not use the model's self-assessment as the only filter: in scenarios where errors are costly, add external verification or a database lookup;
• for Chinese-language tasks, take into account the share of Chinese questions in the test (36%) instead of relying only on English-language metrics;
• look at ECE and Slope as closely as you look at raw accuracy.

The authors call improved WorldVQA results a necessary step for the next generation of AI agents: an agent that misidentifies an object while being confident in itself will make mistakes exactly where it acts independently. The open data and scripts let other labs compare their models on the same scale.