Superintelligence Cannot be Contained: Lessons from Computability Theory

摘要

Superintelligence is a hypothetical agent that possesses intelligence farsurpassing that of the brightest and most gifted human minds. In light ofrecent advances in machine intelligence, a number of scientists, philosophersand technologists have revived the discussion about the potential catastrophicrisks entailed by such an entity. In this article, we trace the origins anddevelopment of the neo-fear of superintelligence, and some of the majorproposals for its containment. We argue that such containment is, in principle,impossible, due to fundamental limits inherent to computing itself. Assumingthat a superintelligence will contain a program that includes all the programsthat can be executed by a universal Turing machine on input potentially ascomplex as the state of the world, strict containment requires simulations ofsuch a program, something theoretically (and practically) infeasible.