OpenAI Safety has introduced deliberative alignment, a new strategy for its o-series models. This approach directly teaches models safety specifications and how to reason over them. The strategy enables models to use chain-of-thought (CoT) reasoning to reflect on user prompts, identify relevant text from OpenAI’s internal policies, and draft safer responses.
Key Points
- Deliberative alignment directly teaches reasoning large language models (LLMs) human-written and interpretable safety specifications.
- Models are trained to reason explicitly about these specifications before generating an answer.
- The o1 model, aligned with this method, uses chain-of-thought (CoT) reasoning to process user prompts and apply internal policies.
- This approach achieves precise adherence to OpenAI’s safety policies without requiring human-labeled CoTs or answers.
- o1 demonstrates improved performance over GPT-4o and other state-of-the-art LLMs across various internal and external safety benchmarks.
- Deliberative alignment addresses issues where models previously had to infer desired behavior indirectly from labeled examples.
- Unlike prior alignment methods such as Reinforcement Learning from Human Feedback (RLHF) and Constitutional AI (CAI), deliberative alignment provides the safety specifications directly to the model.
Context
According to OpenAI Safety, previous LLM safety training methods often resulted in models complying with malicious prompts, over-refusing benign queries, or falling victim to jailbreak attacks. These issues arose because models had to respond instantly without sufficient time to reason through complex scenarios, and they inferred desired behavior indirectly from labeled examples rather than directly learning underlying safety standards. Deliberative alignment aims to overcome these challenges by directly teaching models the text of safety specifications and training them to deliberate over these specifications during inference.
Why It Matters
This new alignment strategy indicates a shift in how AI models are trained to handle safety, potentially leading to more robust and context-aware refusals for disallowed content. Builders and deployers of AI systems may see improved reliability in model responses to complex or adversarial prompts, reducing the risk of unintended outputs.
What To Do
- Note that deliberative alignment is applied to OpenAI’s o-series models.
- Watch for further details on how o1's performance on safety benchmarks translates to practical application scenarios.
- Consider how direct instruction of safety policies could influence the development of future AI alignment techniques.