We Overestimate AI's Ability to Say 'No': Why Refusal Isn't the Answer

Our assumption that AI should refuse dangerous requests is treated almost as an axiom. But MIT Technology Review examines why this faith in safety creates a false sense of security.

History and current state

In 2021, Anthropic set a standard: models should be helpful, honest, and harmless. That meant refusing requests for dangerous actions. But disobedience isn't something machines have naturally. Early OpenAI models "would chat about anything," says Steven Adler, who worked there from 2020 to 2024. Ryan McBain of Harvard recalls that early chatbots readily handed out instructions for suicide.

Today, models refuse countless requests — from building bioweapons to instructions for violence. Before launching ChatGPT, OpenAI hired red teams, including Paul Röttger, a researcher studying online extremism. They bombarded the model with thousands of tests tracked in spreadsheets. When Röttger asked it to write a recruitment post for Al Qaeda, the model complied. Months later, it refused. OpenAI compiled this data into datasets and fed it back into the model through fine-tuning.

How refusal works: what we don't understand

Refusal remains one of the aspects of models that we don't understand in any classical sense. When a model encounters a dangerous request, certain "activations" light up among its parameters, like neurons firing in a brain.

According to a Google-funded study by Jannes Elstner at Apollo Research, these activations look like "high-dimensional polyhedral cones." Andy Arditi showed that if you remove these activations, the model stops refusing. The key problem: even after identifying all the components of refusal, elusive, uncountable elements remain. We can see the refusal happen, but we don't understand how it works — it's like a mechanic telling you that no one really knows what happens when you step on the brake.

Classifiers and the computational cost

Because of this unpredictability, companies surround models with layers of classifiers — smaller models acting like a PR team for a loose-lipped executive. Some block dangerous requests on the way in; others filter harmful responses on the way out. They're stacked like layers of "Swiss cheese" — the idea being that enough overlapping holes eventually form an impenetrable wall.

This demands enormous computational resources. Anthropic has reported that one type of classifier alone adds 24% to compute costs. Companies have recently shifted to more efficient "probes" — classifiers that monitor a model's internal activations, like a functional MRI. But all of these remain probabilistic tools. Ryan McBain found that when the same dangerous suicide-related question is asked repeatedly, models usually refuse — but not always.

The paradox: help and harm are inseparable

Useful and harmful capabilities can't be pulled apart. To help cure cancer, AI needs expertise in genetics — the very same expertise that can be used to build bioweapons. Steven Adler points out that it's impossible to strip all sexualized content out of training data.

On top of that, companies draw the line between acceptable and unacceptable with extreme secrecy. Zico Kolter of OpenAI and co-founder of Gray Swan says: "Where to draw the line is a massive question." The situation gets more complicated as governments begin drawing their own lines. Authoritarian regimes could use the refusal mechanism not to block harm, but to erase criticism and legitimate speech. AI already refuses to comment on certain authoritarian leaders. The Pentagon is fighting with companies over its own refusal rules.

"When refusal works perfectly, it can enable serious acts of repression," the author writes. Or, in the worst case: the machines will start drawing the line themselves.