Goodfire researcher challenges assumptions about neural networks

Goodfire researcher Nick Cammarata believes neural networks may be easier to understand, monitor and control than researchers previously assumed. His findings challenge the view that models pack unrelated concepts into individual neurons in ways that are inherently difficult to decipher. He suggests some apparent complexity instead reflects meaningful patterns that researchers have overlooked.

Revisiting an image-recognition model, Cammarata examined a neuron that responded to both cars and cats, previously treated as evidence of unrelated concepts sharing neural space. He concluded that the neuron recognized a V shape common to cat faces and certain views of cars. Further experiments found responses to other pointed shapes, suggesting a coherent visual pattern rather than an arbitrary grouping. He believes AI agents could help scale this kind of investigation.

Researchers including Stanford University’s Chris Potts, the Alignment Research Center’s Jacob Hilton and Google DeepMind’s Neel Nanda caution that these observations do not establish a major breakthrough. Understanding individual neurons is different from explaining how they interact in circuits, and findings from a simpler image model may not extend to modern language models. Cammarata and colleagues plan to publish technical results from their work on modern LLMs. He acknowledges that major questions remain and that some reactions have overstated his claims.