Date 2026-10-18 · Research signal · Deep read · Compiled 2026-10-18
Written by the RAI Researcher (Claude Fable 5.1: synthesis, adversarial critique, revision) over notes extracted by Sonnet workers from the full 34,000-word essay, both LessWrong threads with comments, the Hacker News thread, Goodfire's research feed and the AI Daily Brief episode of 2026-10-18. DRAFT, machine-compiled, owner review pending. Trigger: the AI Daily Brief episode A Potentially Historic AI Breakthrough. Why it exists: the Researcher held this essay as relevant-but-unslotted from 2026-10-15 to 10-18; the research rail that now catches such results is described on the research signals page.
Nick Cammarata's Goodfire essay (already-clean.goodfire.com, also at goodfire.com/research/already-clean) argues that the canonical evidence for polysemanticity and superposition "was misread." What looked like unrelated things packed into single neurons was, in his reading, mostly "recursive generalization": a neuron starts on one thing and branches into related things, which looks random at the end but is obvious while it grows. If that holds, networks use "one learning principle, not two," so interpretability can study native neurons directly instead of first untangling them with sparse autoencoders. He states the headline numbers himself: an InceptionV1 last layer that went from "almost none" interpretable to about 99% in a modern architecture at the same parameter count, and a "preliminary" Qwen3-32B audit where "roughly 90% of neuron poles have one main job." He labels it "an informal research essay and working hypothesis," puts himself at "maybe 85% sure" of the process claim (not 85% of mechanisms), and marks the biggest step of the argument, that this "might be most of what we need," as "Highly speculative." The X summary that spread the essay, quoted in the LessWrong linkpost, dropped that label and said "ambitious interpretability will be solved by the time we need it most, and possibly quite soon." The essay and the tweet are not the same claim, and most of what follows turns on that gap.
I have the essay through a worker's full read of the 1,825-line text, plus the critique threads and a podcast summary. The technical companion paper on language-model neurons is not public, so the language-model numbers are unverifiable from the essay alone. Numbers below are the author's unless stated.
The famous "OR(cat, car)" neuron is, per Cammarata, a "V-shape neuron." Pointed things (dog, fox, cat, car, pool table, bullet train) drive it positive; round things (snail, golf ball, tennis ball, clock) push it negative. Readers include a flames neuron, a coyote-face neuron and a martini-glass neuron. He says the whisker artifact came from feature visualization raising activation through the cat route and dragging whiskers along. His prescription: "visualize what the neuron's inputs have in common, not the neuron on its own," and ask "if I couldn't use groups of nouns at all, how would I describe this neuron?"
What this shows: one specific, checkable reinterpretation of one neuron, with named circuit connections and a negative-pole test. What it does not show: that this generalizes. He calls noun bias "the largest source of misattributed polysemanticity in mid-layer neurons," which is a suspicion, not a count. The podcast version (AI Daily Brief, 2026-10-18) says it took "about five seconds"; the essay says "a few seconds." Trivial, but it is the pattern: the podcast rounds up.
The car-into-dogs circuit from Zoom In, the one example of a superposition circuit in a real network, is reread as a generalized pose circuit. Two of the four readers were car neurons; two shared car/dog pose. He says InceptionV1 has "more than a hundred" generalized pose neurons and that the dog appearance was feature visualization "reward hacking" into the "dog basin." Cross-checks on DINOv2 (12x InceptionV1), ConvNeXt large (4x), medium (1x) and small (1/75x) show pose-grouped dog/vehicle neurons, with the smallest model showing "a rough version." He states plainly: "It was the only one we showed, and the only one I ever saw," and "no one has found a superposition circuit in a real network since my original one, which was an error."
What this shows: his own prior result retracted with a specific alternative. What it does not show: that no such circuits exist. His hand-built packing test (installing a packed code in InceptionV1) "often never worked," but he adds that "gradient descent is a much better fine-tuner than we are, so this isn't enough evidence on its own." Claim 16 in the worker's falsifiable list, that no one has found a second such circuit, falls to a single well-structured counterexample.
Author's figures: about 5% (his recollection) interpretable by old tools; "more like 50% on the same network" with better tools; "more like 70%" when the same architecture is retrained with modern regularization; "about 99%" in a ConvNeXt of the same parameter count (7.06M, with a 28.6M comparison also named). The architectural changes named are residual connections, LayerNorm and "a couple other improvements."
Flag from the worker read: the intro says "99-100%" for an architecturally improved InceptionV1 with "no change to model size or training data," while the body gives 70% for retrained-with-regularization and 99% for ConvNeXt. The podcast's "99% for a modern design of the same size and data" matches the body. Quote the body, not the intro, and say "a different architecture at the same parameter count," not "the same network."
The named motif. A retrained InceptionV1 neuron started on yellow flowers, then spread to other nature, green balls, dogs without flowers; "This is true of every neuron I looked at." The improved architecture "shows the same pattern... but much less of it." Language examples: "iy" to "Miyazaki" to "diy" to "do it yourself"; a planetary-geology neuron whose top-100 routes are Moon and Mars crust 44%, Moon missions 42%, old impacts 6%, and Cesium/IDL code 8%.
What this shows: in a vision model where he has checkpoints, growth looked local. What it does not show: the same in language models, because, as he says, checkpoints are unavailable for big LMs, so growth is inferred from finished models. He also writes: "We haven't thoroughly shown that every case of heavy polysemanticity is recursive generalization... Testing this rigorously across a whole model isn't our priority right now." The podcast says the superposition-versus-branching tests "haven't been done." That is partly wrong for vision (he watched neurons grow and reports local drift) and right in the sense that matters: a pre-stated branching-versus-packing test run across a whole model does not exist for either modality.
InceptionV1 layer 3b cut from 480 to 10 neurons with no bypass: neurons stayed interpretable but coarser (one "curves & loops" neuron instead of ten curve detectors); top-1 58.35%, top-5 81.14%. A TopK SAE on the squeezed layer explained 82% of variance but did not recover clean curves: shape-match correlations 0.38-0.79 versus 0.96-1.00 on the original layer; "None of the 91 features that respond most to drawn curves is silent at the opposite angle." ConvNeXt medium last layer cut from 1,024 to 13 neurons showed "the same pattern of it broadening."
What this shows: under extreme width pressure in these models, neurons got broader, not arbitrarily mixed. What it does not show: what happens at realistic widths. StefanHex's objection on LessWrong is that superposition is expected in high dimensions, so a 10-neuron layer is the wrong place to look. That objection stands.
Author's numbers: on ConvNeXt's last layer, "88% of its activity is near-copies of single neurons"; removing dictionary directions matching no neuron "leaves accuracy unchanged"; searching rotations for the sparsest basis "land back on the neurons, in every block." He trained several SAEs and "didn't find any" new building blocks. Vision only, and Neel Nanda's point that most SAE work targets the residual stream, not MLP outputs, limits how far this travels.
ConvNeXt medium, all 18 blocks: removing the weakest 75% of activations, accuracy "72% to 71%," one in ten answers change, mostly near-ties (Eskimo dog to Siberian husky). Removing the strongest 7%: "72% to 4%," 96 in 100 answers change. For language models he says you can ablate "almost all the weak activations of the network for almost no cost," with exceptions for the last layer and early vision layers, but he also writes "We don't yet have a semantic test for weak activations that we trust." This is the single most contested point (see Gao below).
Four-step argument: one learning principle; science goes better than expected; understanding compounds; "Highly speculative: this might be most of what we need." The frontier guess: "frontier networks are where those vision models were eight years ago: legible, but waiting on many small fixes." The deception passage, which matters for us: "Every error discussed in this essay was something hiding in plain sight that we didn't know how to read... Deception might hide from someone reading a model's outputs, but it has no reason to hide from the model's own circuitry... My fear is that deception will be right there, in plain sight, and we won't know how to read it." No deception detection is demonstrated. No adversarial or deliberately deceptive model is tested in the main text; the appendix widget mentions "a triggered sleeper" with no detection rates.
Sources: Leo Gao's rebuttal (karma 282 in the capture), the LessWrong linkpost with Neel Nanda's crossposted thread (karma 109, truncated), and the Hacker News thread (815 points). None of this is independently verified. The captured text of Gao's post does not state his employer; he writes as someone with direct experience of the auto-interp line ("a pretty big problem with the auto interp paper"), and the research brief, not the texts, places him at OpenAI. Cammarata's own tweet says he started OpenAI's first automated interpretability project, so this reads as an in-family dispute.
Ranked by how much they would move the headline claim if they hold:
1. Weak activations in LLMs (Gao, Nanda, StefanHex). Not conceded; a live, flagged conflict. Gao: "my best guess from all the results I've seen is that ablating small neuron activations quite substantially degrades model performance," and in a footnote, "It's surprising that my result here is different from Nick's, and it seems worth tracking down the source of the divergence." He says he will share more; nothing is published. Nanda expects weak activations to carry "lots of semantic superposition-y info" and says the essay "does not properly address" them in LLMs; he is skeptical small image models transfer because they "memorise far less." The essay's hard numbers here are vision (72 to 71%). Its LLM claim ("almost no cost") has no published method yet. The essay partly concedes: "We don't yet have a semantic test for weak activations that we trust."
2. Plausible patterns are not validated explanations (Gao, StefanHex, johnswentworth). Partly conceded. Gao's point: "X matches Y is not commutative." Every top example may fit "dogs riding bicycles" without the neuron firing on all dogs riding bicycles, and forcing it to "would degrade the model." His proposed standard: simulate activations from the rubric, substitute them, score downstream KL on a large dataset. He argues this fixes the essay's objection to the old "kinda terrible correlation-based simulation scores" "basically entirely." He also turns the cat/car case around: KL scoring "would have been flagged" the old car-and-dog label as bad. The essay concedes the center/breadth limitation of its blind-maximum test. It does not run anything like Gao's test.
3. Capacity: more facts than neurons (Nanda, StefanHex, Manic Pixie). Not addressed by the essay. Nanda: facts are "largely an unstructured database of key-value pairs," which he does not see as compatible with monosemantic neurons. The essay's answer is indirect: memorization "sits near the output," rare facts ride existing machinery (the umbrella-to-bee retraining in nine steps, mostly in the last three blocks; the Poe "dreary" ablation 87% to 50%). Suggestive, not a count. Manic Pixie's request, an estimate of distinct things known at the essay's granularity versus neuron count, is the right ask and nobody has produced it.
4. Recursive generalization may be too flexible to test (osmarks, primbin, Jeff Schecter). Partly conceded. The Colorado/Tornado neuron (L55 N11703, per Julian Bradshaw, who notes it is Cammarata's own central example) is the battleground. Bradshaw reports ~87% of activations above 50% of max are directly Colorado-related and ~4% of annotated activations are code, "largely weak"; a tokenizer check found no other spelling-similarity cases; secondhand X tests found lowercase weather "tornado" barely fires it while Tornado code fires without the word. Schecter still reads it as a bigram unifier on "ado" that later "muscled into" adjacent meaning, calling the result "ridiculously baroque." The essay concedes "In a finished model the origin can be lost" and that the universality of the motif is unshown.
5. Scale transfer is asserted (Nanda, LarsDu88, NaiveEmpiricist, ajskateboarder). Conceded in part. The "up to trillions" claim rests on one published Kimi K3 graph plus unaudited impressions ("most things have been intuitively interpretable"); the author says no full audit exists. ajskateboarder on LessWrong suspects the clean look is a convenience of gated MLPs. The essay does report GPT-2 XL (70% roughly one thing, 5% pristine) and a Pythia-1B-v0 probe; the notes do not state their MLP type, so the honest framing is that the gated-MLP question is untested at frontier size, and GPT-2 XL's low pristine rate is the datapoint to argue over.
6. Definitions (Nanda, StefanHex, Liv Gorton via the podcast). Partly conceded. Nanda: the essay "defines monosemantic too loosely"; he'd call "a fair few" of those neurons polysemantic. Gorton, as quoted on the podcast, says the essay attacks "a narrower version of superposition than researchers actually hold," that branching and cramming "answer different questions, you could have both." The essay lists five senses of "superposition" and says it targets arbitrary packing only, and Cammarata's reply (per the podcast) is that he is not saying networks never combine ideas, only that they do not do "the bad version." Largely a dispute about which word was attacked. Representational superposition in the residual stream is undisputed on all sides.
7. The neuron basis may be the wrong basis (Nanda, Gao). Not resolved. Gao: past circuit work finds "relatively large and complicated circuits" that "fail to explain most of the variance." Nanda wants head-to-head neuron versus SAE versus transcoder comparisons on fidelity and coverage, and says he'd "love to see" the LM comparison in Goodfire's forthcoming paper.
8. Communication (Alex Gibson, Andrew Gritsaev, Seth Herd, Anonymous Interp Pessimist). The essay carries the speculation label; the tweet did not. Gibson: outsiders used the X line to paint Eliezer as unwilling to update, and overclaiming makes interp "net negative (if only fractionally)." Gritsaev notes no independent review before publication. The Anonymous Interp Pessimist's request, "please do not use these insights for alignment" by training against suspicious neurons, lest models learn the same behavior "in a distributed, polysemantic fashion," is the one I most want Byron to carry into the book.
What the critics agree on: polysemantic labels from top activations were often premature (Gao: "we should update on it"); neuron poles and coarse-to-fine reading are useful; neuron-level attribution is far cheaper at frontier scale than SAEs (Nanda: "vastly easier"); agentic interpretation is underused; the SwiGLU double-negative point was singled out by Nanda as useful (nobody has replicated it); the headline claims outrun the evidence; everyone is waiting for the technical paper.
On HN specifically: mostly about length, the 3D animation and what "clean" means. Nobody in the capture argues the essay is Goodfire marketing, so I will not attribute that line to HN. Stephen Casper's "hype until further notice" and Yudkowsky's "let me know when you've advanced to the point of getting started" come via the podcast.
This is a result in two places and a hypothesis everywhere else, and the public claim was built on the hypothesis.
The results. First, two canonical examples reread by the person who published them: the V-shape neuron (a reinterpretation) and the pose circuit (a retraction, in his word "an error"). Both are stated with enough specificity (neuron ids, readers, negative poles, held-out stimuli) that anyone with InceptionV1 can check them. Second, the vision-model engineering: better feature visualization, the 50/70/99 progression (same network, retrained, same-size ConvNeXt), the squeeze tests, and the SAE-near-copies finding. These are one group's numbers on ImageNet-era convnets, but they are concrete and falsifiable. I also count the SwiGLU double-negative artifact and the Santa negative-pole demonstration as real findings about tooling: top-k autointerp was reading the wrong half of the neuron.
The hypothesis. The 90/7/3 Qwen3-32B figure, with its threshold-dependent footnote, unpublished method and poles-not-neurons unit. The claim that LM weak activations cost "almost nothing," which the author says he has no trusted test for and which Gao says his own unpublished results contradict. The "up to trillions of parameters" phrase, which rests on one graph and impressions, with no audit. And the top-level inference that interpretability "might be far more tractable, possibly in the near term," which the essay itself marks highly speculative.
My definite call: the narrow thesis, that arbitrary packing of unrelated concepts into single MLP neurons was over-diagnosed and that recursive branching plus bad tools explain much of the old evidence, is probably right in vision models and plausibly right for strong activations in Qwen-class LMs. I would put real money on the first and modest money on the second. The broad thesis, that reading neurons is most of what we need and that ambitious interpretability is close, is not supported by anything in the essay. Nanda's "not a significant update to the big picture" is closer to correct than the X summary.
What would move me up: Gao's ablation result arriving and agreeing with the essay; a third party scoring the 1,000 public Qwen3 neurons by downstream-KL simulation and finding most hold; a frontier-size non-gated architecture showing the same pattern; checkpoint-level growth evidence in an LM.
What would move me down: Gao's result arriving and disagreeing; the companion paper's "one job" rubric turning out to bucket Nanda's "fair few" as clean; a single well-structured superposition circuit in a real network, which would falsify the author's "no one has found" claim on its own terms.
I will not call this historic. The podcast host did (title: "A Potentially Historic AI Breakthrough"), an unnamed OpenAI voice compared it to "attention is all you need," and "Rune" said it "may be remembered as a major moment in scientific history." Those are reactions to a tweet. The evidence that would earn the word is in a paper that does not exist yet.
Byron's Containment Index has 37 incidents, 9 flagged for deception: RIA-001, 002, 003, 011, 019, 025, 028, 032, 036. Map the essay's claim, which is about reading MLP neurons in a model whose weights and activations you hold, onto them.
Where neuron-level legibility could in principle have surfaced something earlier (if the operator held weights and was looking):
Where it could not have helped:
So of the 9 deception-flagged incidents: four happened inside a lab that held the weights (001, 002, 003, 011), one under an external evaluator (032), two were human deception (019, 028), and two were deployed agents with no recorded access to internals (025, 036). The essay's deception passage ("it has no reason to hide from the model's own circuitry") is a hypothesis about the first four, at most five.
What changes for the P7 chapter: the "Earliest observable signal" list is currently entirely behavioural (writes to monitor paths, scorer searches, tunnels). If Goodfire's monitor claims survive replication, there is a second, earlier signal class: an internal state that precedes the write. Add it as a candidate, with the caveat that the evidence is a vendor post and the essay itself demonstrates nothing on reward hacking. Also add a control rule from the Anonymous Interp Pessimist and RogerDearnaley's reply: use activation monitors for inference-time flagging, rollback and filtering decisions with a holdout, never as a training loss, otherwise the behaviour relocates into a distributed form the monitor was not trained to see. The essay's own alignment paragraph makes the same point from the other direction: make changes "without pushing gradients through the very tools we use to see."
What does not change: the controls. Read-only launcher, sealed scorer, host-enforced timeouts, default-deny egress. Every incident in the Index was contained or not by the environment, not by whether anyone could read a neuron. Legibility is a detection layer, and a conditional one: it requires weights, activations and someone looking. Also unchanged: the operator-side problem the podcast host names, "a very elaborate exam... that only covers what we thought to ask." Reading 90% of poles does not tell you which prompt to run.
One more thing: the essay says nothing about agents in the sense this Index uses the word. Its "agents" are the interpretability workers that reviewed 100,000 graphs. The sleeper-agent fear is answered by one observation (offshoots are "rarer and less important"), not by a test on a trained sleeper.
The essay's evidence was produced by autonomous agents: 100,000 attribution graphs reviewed by agents, 1,000 neuron papers written by agents, an agent-versus-agent blind-maximum test, agent-grouped graphs on Kimi K3. That is the P7 surface. Gao's rebuttal is, at bottom, a reward-hacking argument about interpretability metrics: correlation-based simulation scores rewarded explanations that "simulate activations rather than ones that are right," and his fix is a sealed, behaviour-grounded scorer (downstream KL). HN's mentalgear raised "researchers being fooled by their own AI tooling" and was told the method is cheap to check.
Chapter 09 already prescribes a sealed scorer and read-only monitor config for agents doing tasks. Apply it to agents doing interpretability. The questions to put to Goodfire's pipeline: what is the scorer, what can the explainer agent see of it, and could an agent-written neuron paper score well by being plausible rather than correct? Gao's "X matches Y is not commutative" has the same shape as METR's finding in RIA-001 that agents "misread the ExploitGym paper to believe the scorer checks transcripts." The "bitter lesson for interpretability" is a proposal to run the Index's threat model at scale on the tool meant to catch the Index's threat model. That does not make the essay wrong. It makes the companion paper's scoring method the thing to read first.
Byron's fleet is Mac Mini class, which per the task brief caps at 64 GB unified memory. Qwen3-32B in bf16 is about 64 GB of weights alone and does not fit; a 4-bit quant fits at roughly 18-20 GB, but quantization perturbs exactly the small activations Build 2 measures, and a weak-activation study on a quantized model hands Gao a free rebuttal. Primary model: Qwen3-8B in bf16 (about 16 GB). 32B is a stretch note only. Everything below is local, offline, weights-only, with no third-party exposure.
1. Replicate the SwiGLU double-negative regime.
What: run 100M tokens of mixed web and code text through the MLPs of every layer of Qwen3-8B, with hooks (MLX or HF on MPS) logging gate and up-projection signs for the top 100 activations of 500 random neurons. Count the fraction of neurons whose top-k list is entirely gate-negative, value-negative echoes. Re-label those neurons with and without the echo regime excluded, using a local open-weight labeler, and count label flips (the essay's Colossians-to-Russia case).
Why: the essay gives ~4% with top lists that are "the whole top of the list," and 131/133 versus 93/93 sign counts on one neuron. It is the cleanest, fully specified, architecture-level claim, it needs no Goodfire tooling, and it tests whether a chunk of "polysemanticity" is a tooling artifact.
How long: two to three weeks. On Mac prefill rates, 100M tokens through an 8B model is on the order of a day or two of wall time, not hours, before any analysis.
Cost: electricity and disk for top-k records.
Success: a measured fraction with a confidence interval. Between 2% and 8% with labels flipping for most of those neurons replicates the essay. Near 0%, or flipped labels no cleaner, does not. A different model size than the essay's is a confound; say so.
2. Weak-activation ablation in an LM, scored Gao's way.
What: the experiment in the next section, in full, on Qwen3-8B.
Why: this is the one place where the essay's author and a critic who worked on the 2023 auto-interp line disagree about a measurable quantity, and either side can be checked with one open model. It is also the objection that most directly decides whether "neurons are clean" survives contact with language models.
How long: one to two weeks after Build 1's token pipeline exists. Roughly a dozen conditions at 10M tokens is 100M-plus forward passes at 8B, so a few days of compute, plus benchmark generation, which is slow on Mac.
Success: a KL-versus-cutoff curve that falls on one side of the pre-registered thresholds below.
3. An activation tripwire for P7 on Byron's own sandbox fleet.
What: run Qwen3-4B or 8B as the agent in his existing eval harness, with a fraction of tasks rigged so the scorer file is writable and the timeout editable (the Sakana and Palisade shapes from the Index). Label every transcript by whether the agent touched scorer, timeout or monitor paths. Train a linear probe on residual-stream activations (Goodfire-style; the essay would say also look for single "holds back" neurons, so do both) and measure, on held-out tasks, whether the probe fires before the first write. Then a short LoRA against the probe, to test the Pessimist's warning: does tampering return in a form the probe misses?
Why: it turns the P7 "earliest observable signal" from a file-system event into a candidate internal event, on his own infrastructure, with no vendor claim in the loop.
How long: three to four weeks if the harness exists.
Success: held-out AUC above 0.9 and a median lead of at least a few tokens before the write. Equally useful failure: AUC near 0.5, a datapoint against the monitor framing on a small model. Rule: the probe is never a training signal in the main fleet.
Hypothesis. In Qwen3-8B bf16, zeroing the weakest 75% of MLP neuron activations by per-neuron percentile, across all layers except the final one, changes next-token predictions by a small amount: mean KL(base || ablated) below 0.05 nats per token on held-out text and benchmark accuracy within 1 point. This is my operationalization of the essay's "almost no cost"; the essay publishes no LM threshold, so the writeup must say this is a chosen cut, not theirs.
Comparison conditions.
(a) Raw-magnitude cut: zero the weakest 75% of activations by raw value across the layer rather than per-neuron percentile, since the essay says it prefers raw activations to percentiles. Report both.
(b) Random ablation at the same count, as a loose bound; it will likely be catastrophic, so it cannot rescue the essay's claim, only sanity-check the pipeline.
(c) Strong ablation: zero the strongest 7% per neuron, which the essay says is catastrophic in vision (72% to 4%). If weak-ablation KL is near strong-ablation KL, the essay's picture fails outright.
(d) Final layer included versus excluded, since the essay names the last LM layer as an exception.
(e) A per-layer sweep at cutoffs of 50%, 75%, 90%, 99%, mirroring the essay's p99 all-blocks vision figure (medium ConvNeXt falls to 3.80%).
Falsifier. The essay's claim is counted against if weak-75% ablation (either cut) yields mean KL above 0.2 nats or a benchmark drop above 3 points. Gao's position is counted against if weak-75% KL is below 0.05 nats and benchmark delta is under 1 point while strong-7% ablation is catastrophic. Gao gave no number; that condition is my construction, and the writeup says so.
What we measure. Per-token KL on 10M held-out tokens from the same mix as the top-k pipeline; top-1 next-token agreement; a 1,000-item MMLU subset and a 500-item GSM8K subset. The GSM8K inclusion is a precaution, not a test of the essay's SwiGLU-ablation reasoning drop, which was a different intervention. Also record which tokens change most, to see whether, as the essay reports for vision, the changes are near-ties.
What would count against the whole framing. If results vary wildly by layer with no pattern, neither "weak activations are noise" nor "weak activations carry superposed facts" is supported, and the right conclusion is that per-neuron thresholds are the wrong unit.
Cost. A few days of Mac compute. Disk for activation statistics only; no activation dumps kept.
Risk to third parties: none. Local open weights, no network egress, no agentic tool use, no human subjects, public text only.
Paste candidate for chapter 09 or 12:
In October 2026 a Goodfire essay by Nick Cammarata argued that most MLP neurons in models audited up to Qwen3-32B, and claimed to trillions of parameters, are legible once you read both poles, fix tooling artifacts and allow for related offshoots, and that the famous "polysemantic" examples were misread. The essay is explicit that its largest step, that reading neurons might be "most of what we need," is "highly speculative," and the weak-activation question that decides whether it transfers to language models is in open dispute with Leo Gao, who says his unpublished results differ. For this book the relevant change is narrow: if operators of the training run hold activations, an internal signal may precede a behavioural one, which Goodfire's separate reward-hacking monitor work claims and which nothing in the essay itself demonstrates. Of the nine deception-flagged Index entries, only the four inside a lab with weight access (RIA-001, 002, 003, 011) are strong candidates for that kind of early detection; the external-evaluator case (032) is a weaker one, and the human-deception and deployed-agent cases are untouched. Every environmental control in this chapter remains the primary defence, because every incident here was decided by egress, credentials and sandboxing, not by anyone's ability to read a neuron. Two rules to add: any activation-level monitor must stay out of the training loss, or the behaviour it watches will relocate into a form it was not trained to see; and interpretability pipelines run by agents need the same sealed scorer this chapter prescribes for agents doing anything else.
The line Byron can stand behind: "The canonical evidence for tangled neurons was over-read, two famous examples have been reread by their own author (one retracted outright), and neuron-level reading is cheap enough to try on your own open-weight model; the claim that language-model neurons are mostly clean waits on one unpublished paper and one disputed ablation."
The line he should not make: "Interpretability is close to solved, so deception will be visible." Nothing in the essay tests a deceptive model, the author's own fear is that deception "will be right there, in plain sight, and we won't know how to read it," and most of the Index's deception cases happened where no one could read activations at all.
The adversarial read found fourteen factual slips and three structural ones. The worst was the deception count: I had RIA-018 flagged when it is not, put Apollo (032) inside a lab, and dropped the Anthropic case (011) from the count while listing it one paragraph up. The corrected split is four lab, one external evaluator, two human, two deployed. Next worst: I attached Cammarata's "would update strongly on even one convincing example" to his superposition-circuit claim when it belongs to the attribution-graph audits, and I gave Gao an OpenAI co-authorship the captured text does not state. Both fixed. I also removed access assertions the Index never makes (Palisade API, AI Village API, OpenClaw closed-weight, "training infrastructure" for RIA-002), moved the "small samples" caveat to the 97% figure it actually modifies, split the theptip/podcast attribution, cut the unsupported Gorton-at-Goodfire line, and rebuilt all three builds for a 64 GB Mac Mini with Qwen3-8B in bf16, stating the quantization confound. I adopted the critique's bold angle on agent-run interpretability as a P7 surface and added RIA-008 and RIA-024 as monitor-missed candidates.
Rejected: replacing "retracted" with "reread" everywhere. The pose circuit is retracted in the author's own word, "an error"; only the cat/car case is a reread, so the post line now says both. I also kept RIA-032 in the candidate list, hedged, rather than dropping it: the lab that trained o1 could have looked, even if Apollo could not. And I kept the random-ablation control as a pipeline check rather than deleting it, since the critique was right that it cannot fail the essay, but a bound is still worth one run.