Anthropic Research published a paper investigating superposition, memorization, and double descent in simple neural networks. This preliminary investigation trained toy models, previously used in research on infinite-data and underfitting, on limited datasets. The study aimed to understand the mechanistic underpinnings of overfitting, particularly its connection to superposition.
Key Points
- Simple neural networks trained on toy tasks can represent more features than they have neurons, a phenomenon called superposition.
- Overfitting in these models corresponds to storing data points in superposition, rather than features.
- Models exhibit two distinct regimes based on dataset size: an overfitting regime, characterized by data point superposition, and a generalizing regime, characterized by feature superposition.
- The transition between these overfitting and generalizing regimes is marked by the observation of double descent.
- The investigation was a preliminary study using extremely simple toy models.
Context
According to Anthropic Research, understanding overfitting is important for mechanistic interpretability, despite it being a central problem in machine learning with limited mechanistic understanding. Previous work has suggested a link between overfitting and the learning of interpretable features. The research posits that superposition could be central to understanding how models, such as language models, verbatim memorize text. A naive approach of dedicating one neuron per memorized sequence is inefficient, making superposition a candidate for handling mutually exclusive cases without interference.
Why It Matters
This research offers a preliminary mechanistic view into how neural networks handle memorization versus generalization. For builders and researchers, understanding the different forms of superposition (data points versus features) and their relationship to overfitting and double descent provides insight into model behavior under varying data conditions. This could inform strategies for designing more interpretable and steerable AI systems.
What To Do
- Note the distinction between storing data points in superposition during overfitting and storing features in superposition during generalization.
- Consider how dataset size might influence the operational regime of a model (overfitting versus generalizing).
- Observe the phenomenon of double descent as an indicator of transition between these regimes.