Anthropic Research has explored a method called Constitutional AI, which aims to train AI assistants to be harmless without requiring human identification of harmful outputs. This approach relies on a set of rules or principles for human oversight. The process incorporates both a supervised learning (SL) phase and a reinforcement learning (RL) phase.
Key Points
- The Constitutional AI method uses a list of rules or principles as the sole form of human oversight.
- The supervised learning phase involves generating self-critiques and revisions from an initial model, then finetuning the original model on these revised responses.
- The reinforcement learning phase, termed 'RL from AI Feedback' (RLAIF), uses a model to evaluate and select better responses from two samples.
- A preference model is trained from this dataset of AI preferences, which then serves as the reward signal for RL training.
- The method aims to produce a harmless yet non-evasive AI assistant that explains objections to harmful queries.
- Both the SL and RL methods can utilize chain-of-thought reasoning to enhance human-judged performance and transparency.
Context
According to Anthropic Research, as AI systems advance in capability, there is an interest in enlisting AI assistance for supervising other AIs. The Constitutional AI method addresses this by enabling AI systems to improve themselves. The supervised learning phase begins with sampling from an initial model. This model then generates its own critiques and revisions of its outputs. The original model is subsequently finetuned using these revised responses. In the reinforcement learning phase, samples are drawn from the finetuned model. Another model then evaluates which of two samples is superior. This dataset of AI-generated preferences is used to train a preference model, which subsequently provides the reward signal for reinforcement learning.
Why It Matters
This research explores a method for controlling AI behavior with reduced reliance on human labeling for harmful content. For builders, understanding such methods can inform strategies for developing AI systems that adhere to safety guidelines through automated self-correction, potentially reducing the manual effort in content moderation and alignment.
What To Do
- Note the two distinct phases: supervised learning and reinforcement learning.
- Observe how AI feedback is generated and utilized in both phases.
- Consider the role of chain-of-thought reasoning in improving transparency and performance.
- Review the stated outcome of training a harmless but non-evasive AI assistant.