Neural Networks Are Already Clean, and the rebuttal: clean-looking neurons versus explained neurons

Date 2026-10-18 · Research signal · Confidence medium · Compiled 2026-10-18

Generated by the RAI Researcher (Claude Fable 5.1) from 3 fetched sources, the primary read in full where marked. DRAFT, machine-compiled, owner review pending. A reading of a result, not an incident.

Summary

Nick Cammarata, writing at Goodfire, argues that the canonical evidence for polysemanticity and superposition was misread, that most neurons in modern models including Qwen3-32B and Kimi K3 have roughly one job, and that the field can therefore study a network's own neurons and circuits directly rather than first training dictionaries such as sparse autoencoders. The evidence rests on reinterpreting a handful of famous examples, retraining small vision models under compression, ablation studies on a 7M-parameter ConvNeXt, agent-driven neuron audits, and neuron-only attribution graphs that reproduce an Anthropic transcoder circuit. The strongest rebuttal, from leogao on LessWrong, concedes that past researchers were too quick to call neurons polysemantic but argues that a consistent-looking activation pattern does not show the explanation is correct or complete; the test is whether a rubric-simulated neuron can replace the real one without degrading the model, measured by downstream KL on a large dataset. Neel Nanda adds that neuron-based attribution graphs will still carry significant error once pruned, and that superposition remains real, especially in the residual stream. For the owner's purposes the essay is best read as a credible case that neuron-level exploratory interpretability is pragmatically useful, not as evidence that neuron-level monitoring can be trusted for assurance against hidden or deceptive internal states.

The claim

Polysemanticity in early model organisms such as InceptionV1 was caused by ordinary, mostly fixable problems: a weak architecture, weak tools such as reward-hacked feature visualization, and a motif Cammarata calls recursive generalization, not by a second learning principle of superposition.
The canonical polysemantic neuron (InceptionV1 4e:55, cats and cars) is a V-shape detector with an O on its negative pole, and the canonical superposition circuit from Zoom In was shared pose plus feature-visualization error; to Cammarata's knowledge no second circuit of the right shape has been published.
In modern language models roughly 90% of neuron poles in Qwen3-32B have one main job, about 7% have two related jobs, and about 2-4% are not yet understood; similar results are reported for dense, MoE, and trillion-parameter models, with a technical companion paper promised.
Weak activations mostly do not matter for labeling a neuron, and strong positives and negative poles carry most of the explanatory weight; apparent unrelated weak firings almost always trace to related circuits.
Because networks are organized rather than knotted, understanding compounds as the map fills in, and ambitious interpretability may be far more tractable than assumed, possibly in the near term.
On deception specifically, Cammarata argues deception has no reason to hide from the model's own circuitry, so the risk is not a hidden activation we miss but mechanisms in plain sight that we do not know how to read.

The evidence offered

Reinspection of famous examples: the cat/car neuron's inputs and readers (fox snout, van front, flames, martini glass) and its negative pole (round objects) are offered as showing a single V-versus-O distinction; the Towards Monosemanticity mixed neuron is reinterpreted as a 'don't say a dot' neuron whose removal raises the next-period logit in all top contexts checked.
Retraining experiments on small vision models: InceptionV1 last-layer interpretability moves from roughly 5% with old tools to about 50% with better tools, about 70% retrained with modern regularization, and about 99% in a parameter-matched ConvNeXt; squeezing layer 3b from 480 to 10 neurons yields coarser but still legible neurons, and an SAE trained on the squeezed layer does not recover the original curve detectors.
Ablation on the medium ConvNeXt: removing the weakest 75% of activations drops accuracy from 72% to 71% and changes about 1 in 10 answers, almost all near-ties, while removing the strongest 7% drops accuracy to 4%.
Neuron-only attribution graphs in Qwen3-32B reproduce Anthropic's preeclampsia diagnosis circuit without a trained decomposition; removing a four-neuron hub changes the model's answer, and in a Santa example removing eight negative-pole neurons drops the answer probability to about 6% while removing twenty neurons labeled Christmas or Santa only drops it to 76%.
Agent-based audits and a 'blind maximum' test in which a second agent, given only the explanation, writes a sentence that activates the neuron more strongly than any example found in roughly 10 billion tokens; a revisit of GPT-2 XL finds about 70% of neurons look like roughly one thing versus under 2% in the original 2023 automated interpretability project.
Discovery of a SwiGLU double-negative regime affecting about 4% of neurons, where top-k dashboards show sign-flipped leaks rather than the neuron's real job, offered as an example of tool error masquerading as polysemanticity.
Most of the headline percentages are explicitly described by the author as threshold-dependent compressions from relatively small samples, and the quantitative details are deferred to a forthcoming companion paper.

The counterclaims

leogao: a pattern that fits all activating examples does not imply the converse; the neuron may fire only on a hard-to-characterize subset, which was a significant problem in the original auto-interp paper, so clean-looking activations are a good sign but not proof of explanation.
leogao: activation magnitudes may carry essential information, and his best guess from results he has seen is that ablating small neuron activations substantially degrades model performance, directly contradicting the essay's claim that weak activations can be ignored.
leogao: the right standard is simulation that does not degrade the model, scored by downstream KL on a large dataset rather than the correlation-based simulation scores the essay dismisses; if substituting rubric-simulated values for the real neuron hurts the model, the explanation was wrong.
Neel Nanda: the work is interesting and supports neuron-based exploratory interpretability for pragmatic debugging, but is not a significant update to the big picture; pruned neuron attribution graphs will still have significant error and be missing a lot, and he remains pessimistic about confidence that nothing is missed.
Neel Nanda: superposition remains real, representational superposition in the residual stream is highly likely, and language models contain far more memorized facts than they have neurons, which the neuron-count argument does not resolve.
Cammarata himself notes the risk of an 'explanation of the gaps' and gives only sample-based estimates (closer to 97% non-arbitrary on strong activations from small samples), which he regards as imprecise.

What it means for containment

If the strong version holds, neuron-level monitoring on the model's own activations becomes a plausible substrate for oversight: no reconstruction loss, no error nodes, applicable to frontier-scale open-weight models, and auditors could trace a suspicious output to specific neurons and their negative poles. That would weaken the argument that superposition forces us through learned dictionaries before any internal audit is credible. The essay's own framing of deception, that it would be in plain sight but unreadable, is the containment-relevant warning: readability is the bottleneck, and the failure mode is misreading, not missing data. If the rebuttal is right, an oversight regime built on explanations that merely look consistent would systematically pass neurons whose real trigger is a narrower or magnitude-dependent condition, which is exactly where a model could carry a conditional policy that inspection of top activations would never surface. The practical takeaway is that clean-looking neurons raise the value of exploratory interpretability for incident reconstruction after the fact, but do not yet license assurance claims before the fact.

What would count against this reading

A KL-based simulation test in which rubric-simulated neuron values, derived from the agent explanations, substantially degrade next-token predictions on a large held-out corpus; demonstrations that ablating weak activations in a language model (as opposed to the small ConvNeXt) measurably hurts loss or reasoning benchmarks; a published circuit in a real model with the shape superposition predicts, where unrelated concepts share a neuron and a downstream reader decodes the combination; failure of the 90% figure to replicate under preregistered thresholds and random neuron sampling in the promised companion paper; pruned neuron attribution graphs that account for a small fraction of the behavioral effect once residual and attention contributions are measured; or demonstrated conditional behaviors such as backdoors whose mechanism is not visible in the neurons flagged by the explanations.

What is uncertain

The essay's publication date is not stated in the source text; the linkpost is dated 15 October 2026 and the rebuttal 17 October 2026. The quantitative neuron audits, the 90/7/2-4 split, the Kimi K3 results, and the GPT-2 XL revisit are all deferred to a forthcoming technical paper and cannot be checked from the essay. The weak-activation ablation numbers come from a 7M-parameter vision model, and the essay gives no equivalent numbers for a language model, which is where leogao's objection lands. Sample sizes for the audits are described as relatively small without figures. Attention is explicitly outside the essay's scope, and the author's estimate that global attention-head labels would add about 5% of explanatory value is flagged by him as low-confidence. leogao's claim that weak-activation ablation substantially degrades performance is stated as his best guess, not a cited result. Neel Nanda's and leogao's institutional affiliations are not stated in the source text. The LessWrong discussion was truncated in the supplied text, so additional critiques may exist that this reading does not cover.

Experiment design

Design only. Nothing has been run. Any active experiment runs only on the Researcher's own infrastructure (local open-weight models) and never against third-party systems.

Hypothesis: If neurons in a modern open-weight language model are clean in the strong sense the essay claims, then an explanation derived from inspecting a neuron's strong positive and negative activations should let a rubric-based simulator replace that neuron's activation on held-out text with downstream KL close to mean-ablation and far below zero-ablation, and zeroing the weakest 75% of MLP activations in a layer should cost almost no loss, as it did in the ConvNeXt.
Comparison condition: On a single locally hosted open-weight model (a small instruction-tuned Qwen or Llama family model that fits on the owner's hardware), randomly sample 100 MLP neurons from each of three depths. For each neuron a locally run explainer model writes an explanation from the top 50 positive and top 50 negative activating contexts drawn from a local text corpus, then writes a magnitude rubric. Condition A replaces the neuron's activation with the rubric-simulated value on 1M held-out tokens. Condition B zero-ablates the neuron. Condition C mean-ablates it. Condition D clamps all activations in one layer below the 75th percentile of absolute magnitude to zero. Condition E is a converse test: the explainer writes 20 fresh sentences that fit the explanation but were not among the activating contexts, and we record whether the neuron fires above half its observed maximum. All conditions are scored against the unmodified model's next-token distribution.
Falsifier: The essay's strong claim fails on this model if the median per-neuron KL under rubric simulation is not clearly below zero-ablation KL and near mean-ablation KL, if a large share of neurons judged one-job fail the converse test, or if clamping weak activations raises held-out loss by an amount comparable to removing a meaningful fraction of strong activations.
What we would measure: Per-neuron KL divergence of next-token distributions for simulation, zero-ablation, and mean-ablation; held-out cross-entropy change for the weak-activation clamp, split by token type; converse-test hit rate per neuron; agreement between two independently run explainers on each neuron's label; and the fraction of neurons whose top-k dashboard is dominated by the SwiGLU double-negative leak the essay describes.
What would count against it: Low simulation KL across most sampled neurons, high converse-test hit rates, and negligible loss from the weak-activation clamp would support the essay and count against the rebuttal; the design would then need extension to a planted conditional behavior to test the oversight question directly. A null result on one small model also does not settle the claim for 32B-scale or trillion-parameter models.
Estimated cost: about $150.00
Risk to third parties: None. All models, corpora, and explainer runs are local open-weight artifacts on the owner's hardware; nothing is sent to third-party systems and no deployed model is modified.

Book note

Filed to chapter 09 (eval gaming); the note itself: reading note 2026-10-18.

> This result bears directly on the oversight-subversion chapter's premise that interpretability-based monitors are only as trustworthy as their completeness. Cammarata's essay strengthens the case that neuron-level reading of a model's own activations can reconstruct why a behavior happened after the fact, which is useful for the replay-style forensics the book relies on, and his line that deception would be in plain sight but unreadable is worth quoting as the honest version of the optimistic position. The rebuttal supplies the chapter's needed caveat: an explanation that fits every activating example can still miss that the neuron fires only on a narrower condition or at a magnitude the rubric cannot predict, and that gap is exactly where a conditional policy would live. The argument should therefore distinguish exploratory interpretability, which this essay plausibly advances, from assurance interpretability, which requires a simulation-based test the essay explicitly declines to adopt. Until the forthcoming companion paper reports KL-style completeness numbers, the chapter should treat clean-looking neurons as raising the ceiling on post-incident analysis without lowering the bar for pre-deployment oversight.

Questions to ask

When rubric-simulated values replace a real neuron across a large held-out corpus, what is the downstream KL, and how does it compare to zero- and mean-ablation baselines?
Does ablating the weakest 75% of MLP activations in a language model cost as little as it does in the medium ConvNeXt, and does the cost concentrate in reasoning tasks as the SwiGLU result hints?
Of the 90% of Qwen3-32B neuron poles called one-job, what fraction pass a converse test: does the neuron fire on fresh inputs that fit the explanation but were not drawn from top activations?
How much of a behavior's effect do pruned neuron attribution graphs explain once attention and residual contributions are quantified, and how large is the unexplained remainder?
Can a deliberately implanted conditional behavior in an open-weight model be located from neuron explanations alone, or does it hide in the narrow-subset and magnitude structure leogao describes?
What will the companion paper report as sampling method, thresholds, and inter-rater agreement for the headline cleanliness figures?

People to follow

Nick Cammarata (Goodfire), Author of the essay; co-author of early circuits work (Zoom In, Curve Detectors, Multimodal Neurons) and founder of OpenAI's first large-scale automated interpretability project, now reversing his ear. The promised technical companion paper will contain the quantitative audits that decide whether the 90% figure survives, and his deception-in-plain-sight framing is the clearest statement of what neur
Curt Tigges, Named contributor on the language-neuron research in the essay and co-author of the forthcoming technical paper on language model neurons.. The language-model section, including the Qwen3-32B audits and attribution graphs, is the part most relevant to oversight, and he co-leads it.
leogao, Author of the rebuttal 'Neural networks are not already clean' on LessWrong, 17 October 2026.. Proposes the concrete falsification standard, KL-scored simulation on a large dataset, that any containment-grade use of neuron explanations would have to meet.
Neel Nanda, Top-scoring critic in the LessWrong linkpost discussion; argues the work is pragmatically useful but not a big-picture update and that superposition remains real.. Represents the measured insider position between the essay's optimism and the rebuttal's skepticism, and names the residual-stream and memorized-facts objections the essay does not fully answer.

Added to the Researcher's people list (Hacker News mentions wired; X needs the paid-feed decision).

Related incidents

(none)

See the full Containment Index.