Anthropic Research has published an analysis detailing six challenges encountered when evaluating AI systems. The report aims to inform those developing AI governance initiatives and organizations focused on AI system evaluation.
Anthropic states that robust evaluations are difficult to develop and implement, and effective AI governance depends on the ability to meaningfully evaluate AI systems. The company uses evaluations to understand its AI systems and improve organizational safety, as detailed in its Responsible Scaling Policy.
The challenges are presented in order from less to more difficult, concluding with policy recommendations.
Key Points
- Massive Multitask Language Understanding (MMLU), a benchmark measuring accuracy across 57 tasks, presents challenges such as potential data contamination, models memorizing answers, and the need for careful judgment calls.
- The Bias Benchmark for Question Answering (BBQ), used to measure social biases in models like Claude, required one of Anthropic's full-time engineers an uninterrupted week to implement and test.
- The BBQ bias score, ranging from -1 to 1, requires nuance and experience to define, compute, and interpret, unlike a simple accuracy metric.
- Anthropic found that some models achieved a BBQ bias score of 0, but were not answering questions, indicating a risk of overinterpreting quantitative scores.
- BIG-bench, a collection of 204 evaluations from over 450 authors, was found to be unwieldy, leading Anthropic to discontinue its use after an initial experiment.
- Stanford's Holistic Evaluation of Language Models (HELM) represents a "top-down" approach where experts curate evaluation tasks.
Context
According to Anthropic, conversations about the societal impacts of AI often revolve around system qualities like truthfulness, fairness, and potential for misuse. The ability to discuss these characteristics relies on technically evaluating models for performance in these areas. However, Anthropic notes that building robust and reliable model evaluations is difficult, and many existing evaluation suites have limitations as accurate indicators of model capabilities or safety.
Why It Matters
This analysis highlights the complexities faced by AI developers and policymakers in creating and interpreting evaluations. It underscores that relying solely on quantitative scores without understanding the underlying evaluation methodology can lead to misinterpretations of model progress and safety, impacting the development of effective AI governance.
What To Do
- Note the specific challenges Anthropic encountered with widely used benchmarks like MMLU and BBQ.
- Consider the time and expertise required to implement and interpret evaluations, as demonstrated by the BBQ implementation for Claude.
- Watch for the policy recommendations Anthropic suggests to address these evaluation challenges.
- Review the Responsible Scaling Policy mentioned by Anthropic to understand their internal safety evaluation practices.