Anthropic Research has published a paper, "Studying Large Language Model Generalization with Influence Functions," detailing a top-down approach to understanding large language models. This work focuses on scaling influence functions, a classic statistical technique, to models significantly larger than previously possible.
Influence functions determine which training examples contribute significantly to a model's outputs. They operate on a counterfactual premise: how would adding a copy of a training example change the trained parameters and, by extension, the model's outputs? The "influence" of a training example approximates its effect on final parameters. This research addresses the computational expense that previously limited influence functions to models with at most hundreds of millions of parameters.
Key Points
- Anthropic's paper, "Studying Large Language Model Generalization with Influence Functions," details a top-down approach to model understanding.
- Influence functions are a statistical technique to identify training examples that significantly contribute to a model's outputs.
- Previous applications of influence functions were limited to models with at most hundreds of millions of parameters due to computational cost.
- New efficient approaches allowed scaling influence functions to large language models up to 52 billion parameters.
- Researchers studied models of 810 million, 6.4 billion, 22 billion, and 52 billion parameters.
- The patterns of generalization become more abstract with increasing model scale.
- Influence patterns typically follow a power law distribution, with a small fraction of training data accounting for most influence.
- Cross-lingual influence strengthens considerably with increased model size.
Context
According to Anthropic, understanding the inner workings of language models has implications for forecasting AI capabilities and aligning AI systems with human preferences. Mechanistic interpretability offers a bottom-up approach, focusing on individual units or small-scale circuits. This research, however, takes a top-down approach, starting with observable behaviors and generalization patterns to investigate the responsible neurons and circuits. The advantage of a top-down method is the ability to directly study high-level cognitive phenomena like reasoning and role-playing that emerge at larger scales. The paper demonstrates efficient algorithms to overcome the computational challenges of inverse-Hessian-vector products and gradient computations, which previously made applying influence functions to large models prohibitively expensive.
Why It Matters
This research provides a method for tracing the origins of model outputs within training data, which can help builders and researchers understand how models generalize. The findings indicate that larger models exhibit more abstract generalization patterns, moving beyond token-level memorization to conceptual relationships. This insight is crucial for developing more reliable, interpretable, and steerable AI systems.
What To Do
- Review the paper to understand the efficient approaches developed for scaling influence functions.
- Note the observed trend that generalization patterns become more abstract with increasing model scale.
- Examine the examples provided, such as the anti-shutdown query and chain-of-thought reasoning, to see how influential sequences differ across model sizes.
- Consider the implications of cross-lingual influence strengthening with model size for multilingual applications.
Keep Exploring
/atlas/claude-family /techniques/system-user-separation /techniques/ptcf