Goodfire Researcher Claims Neural Networks Are Far Simpler Than Interpretability Experts Thought

The main story today isn't another model — it's a debate over how

understandable neural networks really are on the inside

Nick Cammarata, who works at Goodfire (a company specializing in interpretability — decoding what's happening inside neural networks), published an essay with the working title "neural networks are already clean." His thesis: in models with trillions of parameters, the overwhelming majority of neurons are actually readable and comprehensible, if you analyze them correctly.

Where the "tangled neuron" problem came from

For years, one of the main arguments for treating neural networks as black boxes went like this: a single neuron often encodes not one concept but several unrelated ones at once — a phenomenon called polysemanticity. And when several concepts get packed into the same weight space more tightly than there are dimensions to hold them, you get "superposition" — a dense, unreadable tangle. This is exactly the problem that Anthropic cited in last year's work on scaling interpretability methods, and that Goodfire itself referenced in its February piece.

Cammarata draws on personal experience: in 2023 he took part in OpenAI's project on automated neuron interpretation, and back then only about 2% of neurons were deemed humanly readable. That seemed like proof that networks are opaque. But now, Cammarata says, many of the classic "textbook" examples of polysemantic neurons — cited for years as evidence of irresolvable tangling — were simply misread from the start. Under more careful analysis, using AI agents that systematically generate and test hypotheses about what each neuron does, it turns out that most of the network is actually clean and maps onto concepts humans can understand.

Why this matters for AI safety

If the thesis holds up, it strikes directly at the "P(doom)" concerns — the probability of catastrophic outcomes from AI — within the safety community: the more understandable a model's internal structure, the easier it is to verify what it's actually "thinking" and "planning," and the less reason there is to fear hidden, unreadable goals. That's exactly why the post sparked such a strong reaction — the user tszzl and other prominent voices in research circles on Twitter/X called the publication an important moment for the entire field.

What the critics say

Not everyone agrees. Neel Nanda of DeepMind, one of the leading interpretability researchers, said flatly that Cammarata's essay doesn't move his own core assumptions (the "crux" points his worldview is built on): he still expects representations in networks to live in superposition, and believes weak, non-dominant neuron activations also carry meaning — for instance, storing memorized facts that aren't visible under a surface-level analysis.

Leo Gao raised a methodological objection: claims about what a neuron does shouldn't be checked by eye, but rigorously — by "simulating" the neuron and measuring the divergence (KL divergence) between predicted and actual behavior. Jacob Mendel pointed to another weakness: the analysis often looks only at a neuron's strongest activations and ignores the "tail" — weaker but still meaningful firings. And Will Pfau of Anthropic flagged a prosaic but important detail: for now this is an essay, not a peer-reviewed paper with full evidence — it hasn't been published yet.

What this means in practice

For engineers and researchers working on model safety and interpretability, the takeaway is still provisional: there's now a strong argument that agentic analysis methods — where instead of a human manually parsing a neuron, a team of AI agents automatically runs through dozens of hypotheses — could sharply raise the share of "readable" network components compared to the old manual methods from 2023, which topped out around 2%. But rigorous quantitative confirmation — a full paper with metrics like the KL divergence that Gao proposes measuring — doesn't exist yet. So the sensible stance for now is to treat Cammarata's thesis as a promising but unproven hypothesis, and wait for a publication with a reproducible methodology before updating risk assessments or model-auditing approaches.