Berkeley grad students became the AI industry's chief judges: how Arena decides which model is best

While AI companies compete for the title of best model, the power to deliver the verdict has ended up with a group of graduate students at the University of California, Berkeley. Arena, formerly known as LM Arena, has become the main public leaderboard for frontier models. Its rankings are cited in press releases, weighed by venture investors and used by developer teams when choosing a base model. In just seven months, the project has turned from an academic experiment into a full-fledged startup with real influence on the industry.

Arena is built on crowdsourcing: users compare the answers of two anonymous models and vote for the better one. The system accumulates millions of such comparisons and converts them into a rating using the Elo method, the same math that rates chess players. Because the models are anonymous, brand bias is ruled out: users do not know whose answer they are reading until they vote.

The infrastructure that grew out of a university project now affects the biggest players in the market. When OpenAI, Google or Anthropic release a new model, one of the first indicators of success is its position in Arena. Venture funds watch the ranking when making investment decisions. Marketing teams build PR campaigns around a line on the leaderboard.

The system has obvious limitations, though. Voters are an internet audience, not a representative sample of professionals. The tasks users give the models do not always reflect real production scenarios. Finally, Arena's active users are, as a rule, technically savvy enthusiasts, not the average corporate customer.

Still, Arena has filled a vacuum that academic benchmarks could not. Standard tests such as MMLU or HumanEval measure narrow capabilities under controlled conditions. Arena measures what is hardest to formalize: whether people like an answer. That impression ultimately determines which model a user will choose.

Arena's story shows how the academic community can set standards in a fast-moving industry where corporations have neither the time nor the incentive to build neutral evaluation infrastructure. The question is whether that neutrality will last as the startup grows and takes on outside funding.