Humanity's Last Exam: Why Critics Call CAIS's Flagship AI Benchmark a Distraction

Since its release in January 2025, the Humanity's Last Exam (HLE) benchmark has become one of the most discussed tools for evaluating AI systems, and one of the most criticized. KDnuggets analysts gathered a range of expert opinions and concluded that the test distracts the community from what matters more than it offers a useful reference point.

What Is Humanity's Last Exam

HLE was built jointly by the nonprofit Center for AI Safety (CAIS) and the company Scale AI. The benchmark contains 3000 PhD-level questions across more than 100 academic disciplines, including mathematics, molecular biology, classical languages, the history of science and dozens of other fields. Hundreds of professors and graduate students around the world wrote and verified the questions.

Key details:

• Release date: January 2025
• Authors: Center for AI Safety and Scale AI
• Size: 3000 questions, 100+ disciplines
• Best result at launch: about 18% for OpenAI o3
• GPT-4o: about 3%; Claude 3.5 Sonnet: about 8%
• Questions were written and checked by hundreds of scientists

The authors had a clear goal: to show that today's models are still far from the level of human experts on the hardest cognitive tasks. In 2024–2025, public AI demos often created the illusion that AGI was near. HLE became the counterargument: "look how far we still have to go."

Why HLE Is Called a Distraction

Critics' main complaint is irrelevance. The test checks knowledge of rare academic facts: obscure theorems from the century before last, exact quotations from Sanskrit texts, specific biochemical reactions. A low score on such a test does not mean a model is bad at writing code, analyzing data, synthesizing research or helping with medical diagnosis.

The second argument invokes Goodhart's law, well known in science: once a measure becomes a target, it stops being a reliable measure. If leading AI labs begin, openly or quietly, to optimize models for HLE, scores will rise without any real gain in product usefulness. That is exactly what happened with MMLU and several other benchmarks before it.

"We need tests that check how much AI helps me work better, not how

well it knows academic rarities."

The third line of criticism concerns transparency: the HLE questions are kept secret, which makes independent reproduction of results and external audit extremely difficult.

What HLE Supporters Say

Defenders of the benchmark point to its original intent: HLE never claimed to measure product usefulness. Its job is to measure the ceiling of current systems in cognitively hard areas where human expertise has not yet been reproduced. By that standard the test succeeded: it cooled some of the hype and gave journalists, investors and regulators a concrete argument against declaring AGI prematurely.

The creators also note that extremely difficult tests build in a "safety margin." When models start scoring 50–70% on HLE, that will be a genuine alarm signal, not marketing noise.

What This Means

Humanity's Last Exam accomplished its first task: it showed the limits of current AI systems on academically hard problems. As a long-term yardstick of progress, though, it raises legitimate doubts: optimizing for academic rarities does not lead to real usefulness. A useful assessment of AI progress needs benchmarks that test real scenarios: writing code, analyzing data, medical assistance, legal analysis. As long as the choice of yardsticks stays academic, the debate about the "true level of AI" risks spinning in its own vacuum.

Frequently Asked Questions

What score did OpenAI o3 get on Humanity's Last Exam?

According to the January 2025 release, OpenAI o3 got about 18% of answers right, the best result among the models tested at the time of publication. Most other top systems, including GPT-4o and Claude 3.5 Sonnet, stayed in the 3–8% range.

Who created the HLE benchmark and why?

The benchmark was developed jointly by the Center for AI Safety (CAIS) and Scale AI. The authors wanted to show that modern AI systems have not yet reached the level of the best human specialists on hard cognitive tasks, and to cool inflated expectations around AGI.