Artificial superintelligence: Ethics and safety

摘要

While there has been much discussion about the moral status of humans, non-human animals and even other natural entities, discussion about the moral status of digital agents has been limited. This paper proposes a way of reasoning about how we should act towards digital agents under moral uncertainty by considering the particular case of how we should act towards simulations run by an artificial superintelligence (ASI). By placing the problem of simulations within the larger problem of AI-safety (how to ensure a desirable post-ASI outcome) as well as debates about the grounds of moral status, this paper formalizes it into a decision problem. The paper ends by suggesting future steps to solve this decision problem, and how the approach might be generalized.