Flawed AI reasoning is more dangerous than wrong answers

It is widely known that artificial intelligence (AI) still makes mistakes. But a more serious problem may lie in how it reaches its conclusions. As generative AI is increasingly used as an assistant rather than merely a tool, two new studies show that the logic of these models can have serious consequences in critical areas such as healthcare, law and education.

In recent years, the accuracy of large language models (LLMs) in answering questions across a range of topics has risen significantly. That has fueled growing interest in the technology's potential for tasks such as making medical diagnoses, providing therapeutic support or acting as a virtual tutor. Anecdotal reports suggest that users are already widely using off-the-shelf LLMs for such tasks, with mixed success. Recently, a woman in California overturned an eviction notice by using AI for legal advice, but a 60-year-old man was poisoned with bromide after turning to these tools for medical advice. Therapists warn that using AI for mental health support often worsens patients' symptoms.

The new studies suggest that part of the problem is that these models reason in a fundamentally different way from humans, which can cause them to "break down" on harder problems. A recent paper in Nature Machine Intelligence found that models struggle to distinguish users' beliefs from facts, while an unpublished paper on arXiv says that multi-agent systems designed to provide medical advice are prone to reasoning flaws that can derail diagnosis.

"As we move from AI as merely a tool to AI as an agent, the 'how' becomes increasingly important," says James Zou, associate professor of biomedical data science at Stanford School of Medicine and senior author of the Nature Machine Intelligence paper. "Once you use it as a proxy for an advisor, tutor, doctor or even a friend, it's not just the final answer that matters. The whole process and the whole conversation really matter."

Problems in how models make decisions can be especially troublesome in medical settings. There is growing interest in multi-agent systems, in which several AI agents join a collaborative discussion to solve a problem, in the hope of replicating the multidisciplinary teams of doctors who diagnose complex medical conditions, says Lequan Yu, assistant professor of medical AI at the University of Hong Kong. So he and his colleagues set out to examine how these systems reason when solving problems, testing six of them on 3,600 real cases drawn from six medical datasets.

Both groups of researchers say the flaws in the models' reasoning can be traced to how they are trained. The latest LLMs learn to reason through complex, multi-step tasks using reinforcement learning, in which the model is rewarded for logical paths that lead to the correct conclusion. However, they are typically trained on tasks with definite solutions, such as coding and math, which transfer poorly to more open-ended tasks such as identifying a person's subjective beliefs, says Zou.

Focusing on rewards for correct outcomes also means that training does not optimize for good reasoning processes, says Yu. And datasets rarely include the debate and discussion that effective multi-agent medical systems require, which he believes may explain why agents stick to their opinion whether they are right or wrong.

Improving training methods, in particular putting more emphasis on reasoning processes rather than just final outcomes, is a key step. Building datasets that include examples of effective collaboration and debate could also help models develop a more nuanced understanding of complex problems. Only then can we safely rely on AI in critical areas such as healthcare and education.