Responses to catastrophic AGI risk: a survey

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale (‘catastrophic risk’). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs that are safe due to their internal design.

• Publication year

  2014

• Venue

  Physica Scripta

• Publication date

  Unknown publication date

• Fields of study

  Physics, Computer Science, Political Science, Philosophy, Psychology

• Identifiers
  DOI 10.1088/0031-8949/90/1/018001
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Seeing Like a State

  2017cited by this paper
• Robot Ethics: The Ethical and Social Implications of Robotics

  2016cited by this paper
• The Singularity

  2016cited by this paper
• AAAI Presidential Panel on Long-Term AI Futures: Interim Report from the Panel Chairs

  2016cited by this paper
• How We're Predicting AI - or Failing to

  2015cited by this paper
• The machine question: Critical perspectives on AI, robots, and ethics

  2014cited by this paper
• Unfit for the Future

  2014cited by this paper
• Bounding the impact of AGI

  2014cited by this paper
• The singularity and the state of the art in artificial intelligence

  2014cited by this paper
• Universal empathy and ethical bias for artificial general intelligence

  2013cited by this paper
• Superintelligence Does Not Imply Benevolence

  2013cited by this paper
• Framing robot arms control

  2013influential reference
• Consciousness and Ethics

  2013cited by this paper
• Towards Machine Ethics.

  2013cited by this paper
• Shall We Vote on Values, But Bet on Beliefs?

  2013cited by this paper
• Singularity Hypotheses: A Scientific and Philosophical Assessment

  2013cited by this paper
• Arms Control and Intelligence Explosions

  2013cited by this paper
• Ethical machines

  2013cited by this paper
• An Overview of Models of Technological Singularity

  2013cited by this paper
• Implications of a Software-Limited Singularity

  2013cited by this paper
• Omohundro ’ s “ Basic AI Drives ” and Catastrophic Risks

  2013cited by this paper
• Is Brain Emulation Dangerous?

  2013cited by this paper
• Which Consequentialism ? Machine Ethics and Moral Divergence

  2013cited by this paper
• Models of value and choice.

  2012cited by this paper
• On How to Build a Moral Machine

  2012cited by this paper
• The Machine Question: Critical Perspectives on AI, Robots, and Ethics

  2012cited by this paper
• Can Intelligence Explode?

  2012cited by this paper
• Embracing Competitive Balance: The Case for Substrate-Independent Minds and Whole Brain Emulation

  2012cited by this paper
• The Singularity

  2012influential reference
• The Future of Human Evolution

  2012cited by this paper
• Belief in The Singularity is Fideistic

  2012cited by this paper
• Wavelet analysis in neurodynamics

  2012cited by this paper
• Why an Intelligence Explosion is Probable

  2012cited by this paper
• The beginning of infinity : explanations that transform the world

  2012cited by this paper
• Losing Humanity : The Case Against Killer Robots

  2012cited by this paper
• Moral Machines: Teaching Robots Right from Wrong

  2012cited by this paper
• Moral Machines?

  2012influential reference
• WHEN SHOULD TWO MINDS BE CONSIDERED VERSIONS OF ONE ANOTHER

  2012cited by this paper
• EXPERIMENTAL RESEARCH IN WHOLE BRAIN EMULATION: THE NEED FOR INNOVATIVE IN VIVO MEASUREMENT TECHNIQUES

  2012cited by this paper
• On the Feasibility of Internet-Scale Author Identification

  2012cited by this paper
• CogPrime : An Integrative Architecture for Embodied Artificial General Intelligence

  2012cited by this paper
• Moral Machines: Contradiction in Terms or Abdication of Human Responsibility?

  2012cited by this paper
• COALESCING MINDS: BRAIN UPLOADING-RELATED GROUP MIND SCENARIOS

  2012cited by this paper
• Rational Artificial Intelligence for the Greater Good

  2012influential reference
• Leakproofing the Singularity Artificial Intelligence Confinement Problem

  2012cited by this paper
• Human Extinction and Farsighted Universal Surveillance

  2012cited by this paper
• WHY UPLOADING WILL NOT WORK, OR, THE GHOSTS HAUNTING TRANSHUMANISM

  2012cited by this paper
• DIGITAL IMMORTALITY: SELF OR 0010110?

  2012cited by this paper
• The Blank Slate

  2012cited by this paper
• The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents

  2012cited by this paper
• Intelligence Explosion: Evidence and Import

  2012cited by this paper
• Decision Support for Safe AI Design

  2012cited by this paper
• De-anonymizing social networks

  2012cited by this paper
• Advantages of artificial intelligences, uploads, and digital minds

  2012influential reference
• The Singularity and Machine Ethics

  2012influential reference
• A FRAMEWORK FOR APPROACHES TO TRANSFER OF A MIND'S SUBSTRATE

  2012cited by this paper
• Mapping the Landscape of Human-Level Artificial General Intelligence

  2012cited by this paper
• Artificial General Intelligence: 5th International Conference, AGI 2012, Oxford, UK, December 8-11, 2012. Proceedings

  2012cited by this paper
• Super-intelligent machines

  2012cited by this paper
• Thinking Inside the Box: Controlling and Using an Oracle AI

  2012influential reference
• MY BRAIN, MY MIND, AND I: SOME PHILOSOPHICAL ASSUMPTIONS OF MIND-UPLOADING

  2012cited by this paper
• ELECTRON IMAGING TECHNOLOGY FOR WHOLE BRAIN NEURAL CIRCUIT MAPPING

  2012cited by this paper
• Avoiding Unintended AI Behaviors

  2012cited by this paper
• Challenges for Brain Emulation: Why is Building a Brain so Difficult?

  2012cited by this paper
• Safety Engineering for Artificial General Intelligence

  2012cited by this paper
• Motivation Management in AGI Systems

  2012cited by this paper
• Nine Ways to Bias Open-Source AGI Toward Friendliness

  2012influential reference
• The Singularity Institute Whole Brain Emulation and the Evolution of Superorganisms

  2012influential reference
• Delusion, Survival, and Intelligent Agents

  2011cited by this paper
• Mitigating potential hazards to humans from the development of intelligent machines

  2011cited by this paper
• Self-Modi cation and Mortality in Arti cial Agents

  2011cited by this paper
• Artificial General Intelligence - 4th International Conference, AGI 2011, Mountain View, CA, USA, August 3-6, 2011. Proceedings

  2011cited by this paper
• The Beginning of Infinity

  2011cited by this paper
• Model-based Utility Functions

  2011cited by this paper
• Provable Security in the Real World

  2011cited by this paper
• The LIDA Framework as a General Tool for AGI

  2011cited by this paper
• Complex Value Systems are Required to Realize Valuable Futures

  2011cited by this paper
• Artificial Intelligence Safety Engineering: Why Machine Ethics Is a Wrong Approach

  2011cited by this paper
• Race against the machine : how the digital revolution is accelerating innovation, driving productivity, and irreversibly transforming employment and the economy

  2011cited by this paper
• Artificial Intelligence as a Positive and Negative Factor in Global Risk

  2011influential reference
• Machine Ethics: There Is No “I” in “Robot”

  2011cited by this paper
• How long until human-level AI? Results from an expert assessment

  2011cited by this paper
• Bubble Trouble: Off-Line De-Anonymization of Bubble Forms

  2011cited by this paper
• A Family of Gödel Machine Implementations

  2011cited by this paper
• Ray Kurzweil and Uploading: Just Say No!

  2011cited by this paper
• Rational Universal Benevolence: Simpler, Safer, and Wiser Than "Friendly AI"

  2011cited by this paper
• Machine Ethics: Asimov's Laws of Robotics

  2011cited by this paper
• Machine Ethics: Ethics for Self-Improving Machines

  2011cited by this paper
• Machine Ethics: Prospects for a Kantian Machine

  2011cited by this paper
• Incremental Machine Ethics

  2011cited by this paper
• Machine Ethics: The Unacceptability of Asimov's Three Laws of Robotics as a Basis for Machine Ethics

  2011influential reference
• Personal Identity and Uploading

  2011cited by this paper
• Machine Ethics: Why Machine Ethics?

  2011cited by this paper
• Learning What to Value

  2011cited by this paper
• Breaking the Rules to Rise to Power

  2011cited by this paper
• Self-Modification and Mortality in Artificial Agents

  2011influential reference
• How Long Until Human-Level AI ? Results from an Expert Assessment

  2011cited by this paper
• Machine Ethics: Computational Models of Ethical Reasoning

  2011cited by this paper
• Emerging Technologies and the Future of Philosophy

  2011cited by this paper
• Cochlear implants and spoken language processing abilities: Review and assessment of the literature

  2010cited by this paper

• Negotiating Relationships with ChatGPT: Perceptions, External Influences, and Strategies for AI Companionship

  2026cites this paper
• AI going rogue? An integrative narrative review of the tacit assumptions underlying existential AI-risks

  2026cites this paper
• Sociopathic AI and Reliance on Career Advice: A Vignette Study Comparing Human and AI-Generated Recommendations

  2026cites this paper
• There is no such thing as conscious artificial intelligence

  2025cites this paper
• AI Risk Management: A Bibliometric Analysis

  2025cites this paper
• Turing Test 2.0: The General Intelligence Threshold

  2025cites this paper
• The Butterfly Effect of Technology: How Various Factors accelerate or hinder the Arrival of Technological Singularity

  2025cites this paper
• AI safety for everyone

  2025cites this paper
• Complex Adaptation and Permissionless Innovation: An Evolutionary Approach to Universal Basic Income

  2025cites this paper
• Framing Affects Support for the Development of Artificial Intelligence in the United States

  2025cites this paper
• AGI: the illusion that distorts and distracts digital governance

  2025cites this paper
• Confluence of Mind and Machine: Exploratory Analysis of Human Agency and Ethical Governance at Nexus of Brain-Computer Interface and Superintelligent AI-Machines

  2025cites this paper
• The Rise of AI Avatars: Legal Personhood, Rights and Liabilities in an Evolving Metaverse

  2024cites this paper
• Exploring the Socio-Technical Imaginary of Artificial General Intelligence in The Bard Large Language Model: A Narrative Analysis on Perspectives and Dialectics

  2024cites this paper
• Interactive embodied evolution for socially adept Artificial General Creatures

  2024influential citation
• The Elephant in the Room - Why AI Safety Demands Diverse Teams

  2024cites this paper
• A Game-Theoretic Approach to Containing Artificial General Intelligence: Insights From Highly Autonomous Aggressive Malware

  2024cites this paper
• Thought Cloning: Learning to Think while Acting by Imitating Human Thinking

  2023cites this paper
• Developing safer AI–concepts from economics to the rescue

  2023cites this paper
• Legal framework for the coexistence of humans and conscious AI

  2023cites this paper
• AIGC Challenges and Opportunities Related to Public Safety: A Case Study of ChatGPT

  2023cites this paper
• The complex relationship of AI ethics and trust in human–AI teaming: insights from advanced real-world subject matter experts

  2023cites this paper
• The Chai Platform's AI Safety Framework

  2023cites this paper
• AI Safety Subproblems for Software Engineering Researchers

  2023cites this paper
• A democratic way of controlling artificial general intelligence

  2022cites this paper
• Toward a Critical Posthumanism for Social Robotics

  2022cites this paper
• Who's really afraid of AI?: Anthropocentric bias and postbiological evolution

  2022cites this paper
• CyberSoc Framework a Systematic Review of the State-of-Art

  2022cites this paper
• On the Controllability of Artificial Intelligence: An Analysis of Limitations

  2022cites this paper
• Distribution of Forward-Looking Responsibility in the EU Process on AI Regulation

  2022cites this paper
• Optimising peace through a Universal Global Peace Treaty to constrain the risk of war from a militarised artificial superintelligence

  2022influential citation
• AI Risk Skepticism

  2021cites this paper
• Curiosity and other attitudes towards sex robots: Results of an online survey

  2021cites this paper
• Preliminary Observations on AI Regulation

  2021cites this paper
• The risks associated with Artificial General Intelligence: A systematic review

  2021influential citation
• Who is in control? Managerial artificial general intelligence (MAGI) for Football

  2021cites this paper
• Uncontrollability of Artificial Intelligence

  2021cites this paper
• Perspectives about artificial moral agents

  2021cites this paper
• Human Induction in Machine Learning

  2021cites this paper
• The dark path to eternal life: Machiavellianism predicts approval of mind upload technology

  2021influential citation
• MACHINE LEARNING IN CYBER-PHYSICAL SYSTEMS AND MANUFACTURING SINGULARITY – IT DOES NOT MEAN TOTAL AUTOMATION, HUMAN IS STILL IN THE CENTRE: Part I – MANUFACTURING SINGULARITY AND AN INTELLIGENT MACHINE ARCHITECTURE

  2020cites this paper
• Artificial Intelligence and Indigenous Perspectives: Protecting and Empowering Intelligent Human Beings

  2020influential citation
• The AI-Based Cyber Threat Landscape

  2020influential citation
• AI safety: state of the field through quantitative lens

  2020cites this paper
• Lessons for Artificial Intelligence from Other Global Risks

  2020cites this paper
• Dynamic Models Applied to Value Learning in Artificial Intelligence

  2020cites this paper
• Robotopias: mapping utopian perspectives on new industrial technology

  2020cites this paper
• On Controllability of AI

  2020cites this paper
• Intelligent Systems and Methods to Combat Covid-19

  2020cites this paper
• Mediating artificial intelligence developments through negative and positive incentives

  2020cites this paper
• The Achilles Heel Hypothesis: Pitfalls for AI Systems via Decision Theoretic Adversaries

  2020cites this paper
• O papel das emoções no processo de tomada de decisão moral diante de conflitos bioéticos

  2020cites this paper
• Post-COVID-19 and Business Analytics

  2020cites this paper
• To Regulate or Not: A Social Dynamics Analysis of an Idealised AI Race

  2020cites this paper
• AI Development Race Can Be Mediated on Heterogeneous Networks

  2020cites this paper
• Understanding the Determinants in the Different Government AI Adoption Stages: Evidence of Local Government Chatbots in China

  2020cites this paper
• Achilles Heels for AGI/ASI via Decision Theoretic Adversaries

  2020cites this paper
• Artificial intelligence development races in heterogeneous settings

  2020cites this paper
• Predicting future AI failures from historic examples

  2019cites this paper
• Facing disaster: the great challenges framework

  2019influential citation
• Chapter 2 Risks of the Journey to the Singularity

  2019cites this paper
• Modelling the Safety and Surveillance of the AI Race

  2019influential citation
• Artificial Intelligence to Super Artificial Intelligence, Cyber Culture to Transhumanist Culture

  2019cites this paper
• Singularity’s Potential for Sustainability and Environmental Health and Well-Being

  2019cites this paper
• Long-term trajectories of human civilization

  2019cites this paper
• Personal Universes: A Solution to the Multi-Agent Value Alignment Problem

  2019cites this paper
• Modelos dinâmicos aplicados à aprendizagem de valores em inteligência artificial

  2019cites this paper
• The risks of artificial intelligence

  2018cites this paper
• The Key Concepts of Ethics of Artificial Intelligence

  2018influential citation
• Human Indignity: From Legal AI Personhood to Selfish Memes

  2018cites this paper
• Towards Safe Artificial General Intelligence

  2018cites this paper
• Superintelligence Skepticism as a Political Tool

  2018cites this paper
• Why We Do Not Evolve Software? Analysis of Evolutionary Algorithms

  2018cites this paper
• Uploading Brain into Computer: Whom to Upload First?

  2018cites this paper
• Stovepiping and Malicious Software: A Critical Review of AGI Containment

  2018cites this paper
• Disjunctive Scenarios of Catastrophic AI Risk

  2018influential citation
• AGI Safety Literature Review

  2018cites this paper
• Superintelligence and the Future of Governance

  2018cites this paper
• Classification of global catastrophic risks connected with artificial intelligence

  2018cites this paper
• A Cyber Science Based Ontology for Artificial General Intelligence Containment

  2018cites this paper
• The Emotional Nature of Post-Cognitive Singularities

  2017cites this paper
• Superintelligence As a Cause or Cure For Risks of Astronomical Suffering

  2017cites this paper
• The Formalization of AI Risk Management and Safety Standards

  2017cites this paper
• Towards Moral Autonomous Systems

  2017cites this paper
• Liability Law for Present and Future Robotics Technology

  2017cites this paper
• A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy

  2017cites this paper
• Superintelligence and the Future of Governance: On Prioritizing the Control Problem at the End of History

  2017cites this paper
• On the promotion of safe and socially beneficial artificial intelligence

  2017cites this paper
• Algorithmic Iteration for Computational Intelligence

  2017cites this paper
• Guidelines for Artificial Intelligence Containment

  2017cites this paper
• What are the ultimate limits to computational techniques: verifier theory and unverifiability

  2017cites this paper
• How feasible is the rapid development of artificial superintelligence?

  2017cites this paper
• Diminishing Returns and Recursive Self Improving Artificial Intelligence

  2017cites this paper
• Responses to the Journey to the Singularity

  2017cites this paper
• Risks of the Journey to the Singularity

  2017cites this paper
• Computer Simulations as a Technological Singularity in the Empirical Sciences

  2017cites this paper
• The Landscape of AI Safety and Beneficence Research: Input for Brainstorming at Beneficial AI 2017∗

  2017cites this paper
• Taxonomy of Pathways to Dangerous Artificial Intelligence

  2016cites this paper
• On the origin of synthetic life: attribution of output to a particular algorithm

  2016cites this paper
• A model of pathways to artificial superintelligence catastrophe for risk and decision analysis

  2016influential citation