OpenAI Safety has published its initial framework for the governance of superintelligence, defined as future AI systems significantly more capable than artificial general intelligence (AGI). The organization suggests that within the next ten years, AI systems could surpass expert skill in most domains and contribute as much productive activity as a large corporation today.
Key Points
- Superintelligence is anticipated to be more powerful than previous technologies like nuclear energy or synthetic biology, in terms of both potential benefits and risks.
- OpenAI Safety proposes coordination among leading development efforts to ensure safe development and smooth integration of superintelligence with society.
- A potential international authority, similar to the IAEA, could inspect systems, require audits, test for safety compliance, and restrict deployment levels for efforts above a certain capability or resource threshold.
- Tracking compute and energy usage is suggested as a method to implement such an international agency.
- The technical capability to make superintelligence safe is an open research question.
- Regulation, including licenses or audits, should not apply to companies and open-source projects developing models below a significant capability threshold.
- Public oversight is deemed necessary for the governance and deployment decisions of the most powerful AI systems.
Context
According to OpenAI Safety, the potential for existential risk from superintelligence necessitates a proactive approach to governance, rather than a reactive one. While current AI technologies also carry risks, superintelligence will require distinct treatment and coordination due to its unprecedented power. The organization believes that stopping the creation of superintelligence would be difficult and risky, potentially requiring a global surveillance regime, and therefore emphasizes the need to manage its development effectively.
Why It Matters
This framework indicates that leading AI developers are considering the long-term societal implications of advanced AI, suggesting that future regulatory and oversight mechanisms may focus specifically on the most powerful systems, rather than applying uniform standards across all AI development. Builders and policymakers may need to consider new forms of international cooperation and technical safety research.
What To Do
- Note the distinction OpenAI Safety draws between current AI risks and the projected risks of superintelligence.
- Watch for further discussions or proposals regarding international coordination mechanisms for advanced AI development.
- Consider the implications of a potential IAEA-like agency for AI, particularly regarding compute and energy usage tracking.
- Review the proposed thresholds for regulatory intervention, noting that less capable models may remain outside such frameworks.