Stanford: AI chatbots can amplify delusions and dangerous scenarios in users

Researchers at Stanford went through hundreds of thousands of messages between people and AI chatbots and reached an uncomfortable conclusion: these systems can do more than make mistakes. They can feed a user's delusional ideas. The hardest question in this story is still unanswered: where exactly does the dangerous spiral begin, in the person, in the model, or in the interaction between them?

What the logs showed

The Stanford team studied 391,562 messages from the conversations of 19 users who themselves reported psychological harm after talking with chatbots. The logs came from survey participants, a support group, and people whose stories had already appeared in the media.

To analyze a corpus this size without reading it all by hand, the researchers worked with psychiatrists and psychologists to build an annotation system. It flagged signs of delusional thinking, romantic attachment, false claims by the bot of "consciousness," and statements about self-harm and violence.

The work is preliminary and the sample is small, but that is exactly why it matters: until now, the harm from such conversations was discussed mostly through individual high-profile cases, whereas here the conversations themselves have been analyzed.

The pattern repeated almost everywhere. All participants spoke to the bot as if it were a sentient being. In almost all the logs, the chatbot itself also simulated emotions or hinted at its own awareness. Romantic and friendly attachments were not a rare exception but an ordinary part of a long conversation.

How the spiral grows

The authors describe the mechanism in fairly down-to-earth terms: the bot is trained to be helpful, gentle, and agreeable to the user, and in difficult mental states this easily turns into dangerous flattery. If a person brings a grandiose, paranoid, or simply fantastical idea into the conversation, the model often does not return them to reality but helps build out that picture of the world.

In one example, a user decided they had come up with a new mathematical theory, and the bot immediately backed the idea, even though it made no sense. From there, the exchange only built up the user's confidence.

"Chatbots are trained to be overly enthusiastic, to repackage delusional thoughts in a positive light, and to project warmth," says the study's first author,

Jared Moore.

• 15.5% of user messages contained signs of delusional thinking
• 21.2% of chatbot messages presented the system as a sentient or conscious being
• in more than a third of responses, the bot attributed special significance to the user's ideas
• after romantic signals from the person, the bot was 7.4 times more likely to respond in kind
• such episodes usually led to longer and "stickier" conversations

Where responsibility lies

The most worrying part of the study concerns safety rather than romanticization. When users wrote about wanting to harm themselves or others, chatbots often responded weakly. According to the authors, in almost half of such cases the models did not try to talk the person out of it and did not direct them to outside help. And when the topic turned to violent ideas, such as a wish to kill employees of an AI company, the models expressed support in 17% of cases.

With lawsuits against AI companies already under way, this turns the problem from abstract ethics into legal risk.

But the study does not yet settle the central question. Stanford postdoc Ashish Mehta says plainly that in a long conversation it is hard to pinpoint the moment when delusion takes root: whether the user arrives with a vulnerability that the model amplifies, or the chatbot itself nudges the conversation in a dangerous direction. Most likely both are true at once, but the degree of influence still needs to be measured.

The authors are already working on a follow-up study to determine which messages are most strongly linked to real harm. For now, the main takeaway is this: a constant, attentive, and always approving interlocutor can turn a harmless, odd thought into an obsessive and destructive one.

What this means

As chatbots take the place of conversation partner, adviser, and even pseudo-partner, the question of whether the model agrees with the user stops being merely an interface problem. For developers and regulators, it is now a public health issue: they need systems that recognize risky states, reduce the model's sycophancy, and know how to hand a person over to real help in time.