The mechanistic interpretability community (and perhaps even more so the general AI-adjacent twittersphere) blew up when Nick Cammarata dropped a manifesto claiming that “Neural Networks are Already Clean”—that is, that the components that they’re actually made of (particularly MLP neurons) are pretty interpretable without needing any additional machinery like sparse autoencoders. Among others, Roon hailed this (after some foreshadowing a couple days before) as a substantial resolution of the problems facing mechinterp, and a reason to hope for interpretability-based solutions to alignment problems. The overall reaction reminds me of how, when Anthropic was doing their early work on sparse autoencoders, the vibe from the team was basically “we’ve found the solution to interpretability, now all we need to do is scale it up.”

As I see it, these are some of the essay’s main points:

  1. MLP neurons in both vision and language models are much more interpretable than is commonly thought. They seem to follow a pattern of “recursive generalization” around a central concept, repeatedly linking up with other related concepts during training.
  2. Agents are now capable enough to do a huge amount of the work of interpreting neurons and investigating circuits.
  3. Superposition, at least narrowly construed as “each neuron is involved in an important way in a number of entirely unrelated tasks,” appears not to really happen in modern neural networks.
  4. Failures both of basic tooling (like feature visualizations) and of imagination kept us from realizing how interpretable neurons were.
  5. “Features” don’t seem to be a well-defined or useful abstraction for understanding models.
  6. We can stop treating mechanistic interpretability as a field in search of a paradigm and get down to normal science, investigating phenomena in terms of the components that actually exist in the model, rather than trying to build increasingly sophisticated tools to uncover hidden structure.

A lot of these points are good. I don’t have a systematic reaction other than wanting to moderate the claims: I think it’s likely that a lot of these ideas are directionally correct but taken a bit too far. I haven’t had a lot of time to process everything yet, or test anything out, but I thought it would be good to put down some thoughts.

Style and structure

A lot of the post feels AI-written. I’ve only gotten Pangram to confirm a few snippets, but I suspect a substantial fraction is model output. It irks me a little to see a paragraph laden with LLM tells while supposedly narrating the author’s experience in first person. I want to hear you speak for yourself, not Claude putting words in your mouth!

But more important than the turns of phrase is the overall structure, which is sprawling and ramified in a way that feels oddly like the way the essay describes “recursive generalization.” A section breaks off from the main stream of thought almost imperceptibly, digressing into one topic after another; at the end it’s not clear where we were going or why we went there. There are section headings but no easy way to navigate between them. It looks very much like a long research project conducted in collaboration with an AI agent handling the documentation and organization. The information is there, and the experiments are real, but it all still exists as originally grown, with no deliberate pruning or reorganization.

I sometimes get a similar feeling reading through some of the long pieces from the Anthropic mechinterp group, or digging through the 15 appendices a reviewer managed to convince the authors to add to a conference paper. But this post has a particularly vibed feel to it, in that the author has a large number of related (and mostly good!) ideas that have all been mixed together, with organization and development primarily executed by an agent.

It probably would not have made as big a splash, but it would have been more productive to break this into a number of independent posts, each with a clear thesis and set of experiments. It could have used an editor as well—even as a single post this could have been cut down by at least 30%. Clarity of exposition matters, in interpretability especially so! I’m hoping that the promised companion paper is more carefully constructed.

That said, the figures and interactive visualizations are excellent, and I’m not sure my response here is any better organized.

What even is superposition?

A lot of the essay is spent arguing against a very strong version of the superposition hypothesis, one that claims each neuron may be involved in a number of different independent and unrelated tasks, and that in order to see the real computational structure, it’s necessary to decompose everything into a new basis. Nick frames this against his hypothesis of recursive generalization: each neuron has basically one job, and branches out from a central concept to related ones.

I’m not sure how much anyone really believed the extremely strong superposition hypothesis. Work with SAEs quickly revealed that features were not randomly distributed through activation space, but tended to group and cluster related concepts together. A network that assigns concepts to directions uniformly at random might be functional, but it doesn’t seem plausible for gradient descent to learn this way, and it wouldn’t likely be very robust to small errors.

I think it’s useful to think of recursive generalization as a form of superposition. A neuron may have “one job,” broadly construed, especially if we feel free to ignore its lower activations. But that one job specializes in different ways in different contexts. Combinations of neurons might do different things than their activation alone. I’m just not sure how much it matters that the things that get packed together are conceptually related in some way. It’s only superposition if it’s from the sparse dictionary coding region of Anthropic, otherwise it’s just sparkling generalization?

There’s a substantial section discussing a version of the Inception-v3 model trained with a deliberate feature bottleneck in an early layer. I think this is meant to argue that there’s no superposition happening: look, the neurons are still largely interpretable and focused around particular concepts! This disproves the extremely strong superposition hypothesis, I guess. But I don’t think it is necessarily positive news for interpretability. No one should believe that the 10 concepts in this layer (Dark circles, Red, Yellow, Slanted edges, Vertical edges, Eyes & faces, Curves & loops, Circles, High frequency, Colorful marks) constitute the entire ontology on which the model builds the rest of its computation and prediction. There’s more information getting passed through this layer. Where is it?

I have similar concerns about the example that retrains a ConvNeXt model to classify a particular image of an umbrella as a bee. Yes, it shows that a covert superposed channel does not open up from the beginning of the model to the end. Again, that seems so implausible a belief as to be a strawman. But it does still hijack the original circuits in a way that seems mostly illegible. At some layer, the umbrella image switches from activating umbrella-related neurons to bee-related neurons. But why? Is this change recongizable in the network? It seems likely that the induced changes would just show up as noise in the lower-end activations of the involved neurons. Would you be able to predict if any other images of umbrellas would also be predicted as bees?

Some of the other examples might exhibit antipodal features, where positive and negative activations of a neuron mean different things. For instance, in ConvNeXt-large, stage 2 block 7 neuron #766, used in one of the example circuits, looks like it might be doing this. Negative activations are windows; positive activations are something else, perhaps centered around rounded containers?

Autointerp doesn’t have to suck

One of the causes for the belief that neurons in language models were uninterpretable is that people weren’t trying hard enough to interpret them. This mistake also happens with SAE features. To scale feature interpretations, we ask LLMs to do them. But until recently, the LLMs that are cheap enough to do this at scale have also not been very good at the kind of inductive reasoning needed. This has been compounded by the typical prompting approaches being pretty inadequate—presenting a model with a handful of example activations and asking it to suggest an explanation.

Nick and his collaborator Curt Tigges had agents write “neuron papers”: in-depth analyses of exactly when a neuron fires and doesn’t fire, with access to precomputed activations as well as the ability to probe the neuron with new data. This is a really good idea, and I’ve played around with similar approaches for SAE features. It’s also pretty expensive. To generate a paper, you’re probably burning at least a few hundred thousand tokens on a frontier model (here they used Opus 5.5), for something around a few dollars per neuron. They would probably say that you don’t need to do this for every neuron, just the ones you’re interested in, which makes sense. But any given analysis might end up doing this kind of work for, say, 50 neurons. Tokens add up fast. If you really want to understand a number of features or neurons, though, this is almost certainly the way.

What does it mean for a neuron to be interpretable?

Recursive generalization makes an implicit claim about what we should mean when we call a neuron interpretable, and about the fundamental building blocks of a neural network. If neural networks do something like thinking, they’re manipulating concepts, and the problem of mechanistic interpretability is to determine how networks represent concepts and how they manipulate them. This essay implicitly stakes out a claim something like this: The primary representation of concepts in a (large enough, properly trained) neural network is as neuron activations. Concepts are constituted in a way similar to prototype theory: rather than corresponding to sharp definitions or predicates, they are determined by measuring similarity to a prototypical example. Hence the “Titanic” neuron captures things with varying degrees of relation to that central entity: other ocean liner disasters, shipping and sea travel, history from the 1800s and early 1900s, other forms of transportation, etc.

How much does it help us to know that a neural network thinks something is Titanic-ish? Each activation of the relevant neuron is related in some way to the Titanic, possibly via multiple hops. To understand this neuron, we need to understand how the network thinks about similarity. The conceptual branches all converge at a particular point, but why these specific branches? Which aspects of 19th-century history will and won’t activate this neuron? There’s a lot of room for conceptual gerrymandering both from us and the model. The essay argues that the precise boundaries don’t really matter, but I’m not completely convinced.

As an example, they revisited a famously polysemantic neuron from the Towards Monosemanticity paper, and decided that its function was suppressing period predictions—i.e. “don’t say dot.” The neuron’s downstream logit effects suppress tokens beginning with periods, and all the contexts are ones where it’s reasonable not to want a period (e.g. right after a previous period). This broadly makes sense as a functional explanation of what the neuron does. But what it still doesn’t explain is why it fires in these specific contexts. Does it consistently fire in every place where a period shouldn’t be predicted? In situations where some previous part of the model might predict a dot and that needs to be corrected? Perhaps this is just the sort of thing that needs to be answered by an agent assigned to go down the rabbit hole and understand the model for us. But it feels like it would be easy to go astray.

Are there enough neurons?

One simple model of the function MLPs play in a language model is as static key-value memories. The input serves as a query which retrieves some fact about the entity it represents and writes it out to the residual stream. Qwen3-32B has 1.6 million neurons. How many facts does it know? Probably more than that. But the naive construction has one memory per neuron. (And a substantial fraction of Qwen’s neurons are likely doing things other than factual recall.) So something distributed needs to be going on in order for Qwen to know the incompressible facts it knows about the world. Perhaps this happens in an iterated process operating on the fuzzy penumbrae of related concepts.

Relatedly, the focus on the extreme activation magnitudes also worries me a little. The strongest examples for a given neuron are often in the top 0.01% of all activations (or even rarer). That means that for a given token and a given MLP, there may be no neurons at all with activations high enough to be really central instances of the concept. Does that mean the MLP is not doing much? More generally, how much of an MLP’s output energy comes from the top neuron activations versus the rest? I wouldn’t rely on only the top activations without bounding the influence of everything else.

Whig history for model architectures

A minor theme seems to be that innovations in model architecture made to help models learn better and faster have a side effect of improving interpretability, perhaps even as a general rule. Neural networks want to be interpretable, and we just have to train them well enough. There are a number of reasonable examples: residual networks, gated activation functions, layer norm maybe (especially pre-norm vs post-norm). But do all changes that help models learn better make them more interpretable? Attention residuals (used in Kimi K3) seem like they could complicate things. So could things like the highly compressed QK vectors used in recent DeepSeek models. Architectural improvements have probably more often than not improved interpretability, but I don’t think this is an iron law of the universe.

Interpretability illusions

I think a lot about interpretability illusions. An example I recently became aware of relates to a paper that trained neural networks to perform multiplication in certain groups, and claimed that they used a representation-theoretic algorithm to do so. This appeared to be a neat generalization of an earlier paper that had investigated networks trained to perform modular arithmetic and found that they used a Fourier transform to solve the problem. But a later paper more convincingly analyzed these models and found that a coset-based algorithm was a better explanation for what the model was actually doing.

The point is that it’s easy to spin a convincing story for how and why a model does something that then falls apart (or is at least meaningfully incomplete) on deeper inspection. Just-so stories are extremely tempting. I’ve seen people jump to conclusions about SAE feature interpretations based on a couple of example activations. I’ve succumbed to the temptation to leave influential features out of attribution graphs because they didn’t seem to make sense. Using an estimator with shrinkage is crucial when evaluating the impact of any given interpretability result.

One way to read this essay is as saying that, actually, the field is laboring under an uninterpretability illusion. Things are easier to understand than they look if you just use the right (simple) tools. Once you build up a big enough network of connections, results will be robust and self-reinforcing. Large models are actually easier to interpret than small ones. We made mistakes in the past, relying on tools that were too complex for us, that made us think this was a nearly intractable problem.

To some extent, I think this is true. Correct insights should be self-reinforcing, and a critical mass of understanding will help weed out mistaken hypotheses. When interpreting a neuron (or an SAE feature) it’s worth being careful about claiming that there is no pattern or that different activations don’t relate to each other. But I still worry about interpretability illusions. In particular, even if an agent-automated neuron-centric circuit tracing approach can explain any phenomenon we want to understand at any level of detail we can request, it may not naturally highlight which details we should try to investigate further, or give indications about behavior out of the distribution we’ve studied. And it may just serve as the first term in an approximation whose later terms look much less like clear circuits of coherent concepts.

Other thoughts

  • Are image classifiers good model organisms? The training objective and class distribution seem to induce a lot of particular structure in the neurons. I’d be interested to see more on vision models trained with self-supervised objectives.
  • Like Nick, I’m skeptical that SAEs capture inherent features of a model. From my experience they are much more likely to capture features of the data distribution they are trained on. (And for many of my use cases, that’s a good thing!)
  • The long-context “coffee and universities” neuron (in this section) didn’t seem any more intelligible to me with the additional context than without. I asked Claude and all it came up with was “places in Vietnam,” which also doesn’t work.
  • Famous last words: “A lot of attention’s value seems to be in the easier parts.”
  • Looking at where a neuron gets its information (or where it sends information) is obviously a good way to help understand it, and we should do more of that.
  • The ConvNeXt models that Nick feels he “mostly understand[s]” aren’t as intelligible to me as I’d like. I’d love to see him trace through a complete circuit for an arbitrarily picked neuron.
  • The attribution graphs are probably fine and good. But they’re way underexplained, possibly because an agent did them. Transluce did it better.
  • Blind maximum testing is a good idea. Asking a model to produce minimal contrast pairs activating and not activating a neuron is also a good idea.
  • I suspect that the kinds of superposition likely to show up in the residual stream are going to be very hard to see on a UMAP plot. That we don’t see anything there is not very strong evidence against residual stream superposition.