A recent paper published on arXiv cs.CY critically examines the premise that general intelligence is inherently valuable. The authors present an evolutionary argument, suggesting that the adaptive value of general intelligence is not empirically established over evolutionary timescales.
Key Points
- The paper challenges the assumption that general intelligence, as seen in Homo sapiens, is extraordinarily valuable.
- It notes that many taxa, such as cyanobacteria and horseshoe crabs, have persisted for hundreds of millions or billions of years without general intelligence.
- Homo sapiens has existed for approximately 300,000 years and faces self-generated existential risks.
- Mass extinction events do not show a preference for cognitively sophisticated species.
- The paper argues that general intelligence might be the sole biological strategy that creates existential threats for the species that possess it.
- This concept is termed an "existential risk paradox," which the authors state has no parallel in non-intelligent survival strategies.
Context
According to the authors, the pursuit of artificial general intelligence (AGI) is founded on the belief that general intelligence is highly valuable. The paper scrutinizes this belief by comparing the longevity of species with and without general intelligence. It highlights that species lacking general intelligence have demonstrated remarkable persistence over vast evolutionary periods, while Homo sapiens, possessing general intelligence, faces unique self-generated risks.
Why It Matters
This research prompts builders and researchers to consider the foundational assumptions underlying AGI development. It suggests that the pursuit of general intelligence might introduce unique risks not present in other forms of intelligence or survival strategies, shifting the focus from external AI risks to potential inherent risks of the intelligence itself.
What To Do
- Read the full paper on arXiv to understand the evolutionary arguments in detail.
- Note the specific examples of long-persisting taxa cited in the paper.
- Consider how the "existential risk paradox" might influence discussions on AI safety and alignment.
- Reflect on the implications for the design principles of future AGI systems.
