Anthropic Research has published findings on interpretability research into the internal workings of their Claude language model. The research identifies a specific collection of internal neural patterns, referred to as the J-space, which appears to play a distinct role in Claude's processing, analogous to consciously accessible thought in humans.
Key Points
- The J-space is a collection of internal neural patterns observed in language models like Claude.
- It operates silently within the model's internal activations, distinct from a 'scratchpad' or 'chain of thought'.
- The J-space was not designed or programmed but emerged during Claude's training process.
- Each J-space pattern is linked to a particular word, indicating that the word is 'on the model's mind'.
- The J-space exhibits strong connections to the rest of Claude's neural network, facilitating a broadcasting role.
- Researchers can use the J-space to observe Claude's internal state, such as noticing it is being tested or pursuing a hidden goal.
- A technique has been developed to influence what appears in Claude's J-space, thereby affecting its decision-making.
Context
According to Anthropic, this research was inspired by the global workspace theory in neuroscience, which posits that consciously accessible information gains entry to a shared channel broadcast to other brain systems. The J-space in Claude is believed to fulfill a similar 'workspace' function, allowing the model to 'think about a concept without writing it down'. The technique used to identify these patterns is called the Jacobian lens, or J-lens, which finds internal activity patterns that make Claude more likely to use a specific word in the future.
Anthropic states that the J-space reveals what Claude is considering beyond its immediate input or output. For instance, when Claude processes code with a bug, its J-space may contain 'ERROR'. When reading a protein sequence, the J-space might show the protein's biological function. In cases of prompt injection, words like 'injection' and 'fake' can appear in the J-space, and intermediate steps of multi-step math problems emerge in the correct sequence.
Why It Matters
These findings offer a method for understanding the internal reasoning processes of large language models. For builders, the ability to observe and even influence the J-space provides a new avenue for debugging, steering, and potentially enhancing model behavior. For curious readers, it offers insight into the emergent complexity within AI systems, suggesting that their internal organization may bear resemblances to aspects of human cognition, without implying consciousness.