Anthropic Research has investigated the capacity of language models to evaluate the validity of their own claims and predict which questions they can answer correctly. The research demonstrates that larger models exhibit good calibration on various multiple-choice and true/false questions when presented in an appropriate format.
The study explores self-evaluation on open-ended sampling tasks by having models first propose answers and then evaluate the probability "P(True)" that their answers are correct. This approach yielded encouraging performance, calibration, and scaling across diverse tasks. Self-evaluation performance further improved when models considered multiple self-generated samples before predicting the validity of a specific possibility.
Key Points
- Larger models are well-calibrated on diverse multiple-choice and true/false questions when provided in the correct format.
- Models can evaluate the probability "P(True)" that their proposed answers are correct on open-ended sampling tasks.
- Performance, calibration, and scaling for P(True) are encouraging across a diverse array of tasks.
- Self-evaluation performance improves when models consider many of their own samples before predicting validity.
- Models perform well at predicting "P(IK)", the probability that "I know" the answer, without a proposed answer.
- P(IK) predictions partially generalize across tasks but struggle with calibration on new tasks.
- Predicted P(IK) probabilities increase with relevant source materials in context and hints for mathematical problems.
Context
According to Anthropic Research, the study aimed to understand whether language models could assess the reliability of their own outputs. The methodology involved two primary self-evaluation metrics: "P(True)" for assessing the correctness of a specific answer, and "P(IK)" for predicting the model's general knowledge about a question. The research notes that while P(IK) shows partial generalization, its calibration on new tasks presents a challenge.
Why It Matters
This research offers builders and researchers insights into developing more reliable and transparent AI systems. The ability of models to self-evaluate their confidence and knowledge could lead to systems that are more honest about their limitations, reducing the risk of presenting incorrect information as fact.
What To Do
- Note the distinction between "P(True)" (answer validity) and "P(IK)" (knowledge probability) in model self-evaluation.
- Consider how providing relevant source materials or hints can influence a model's predicted knowledge probability.
- Watch for future research on improving the calibration of P(IK) when models encounter new tasks.