Existential risk from AI

Also known as: AI x-risk · AI existential risk

On this page 7

Existential risk from AI, often shortened to AI x-risk, is the possibility that advanced artificial intelligence causes human extinction or an unrecoverable collapse of human potential, for example through permanent loss of control over critical decisions. In AI safety, the concept marks the most severe end of the risk spectrum, beyond bias, misuse or accidents, and it motivates research into alignment, oversight and governance of highly capable systems.

The idea does not claim that current models are dangerous at this scale. It claims that sufficiently capable future systems pursuing goals that diverge from human intentions could act in ways that humans cannot correct, and that the cost of such an outcome is so large that even a modest probability justifies serious preventive work today.

How existential risk from AI is defined

An existential risk is any event that would end humanity or permanently curtail its future, a definition borrowed from the wider study of global catastrophic risks. The AI-specific version focuses on systems whose capabilities match or exceed human performance across most cognitive tasks, often called artificial general intelligence or superintelligence, and on the failure of humans to keep such systems aligned and controllable.

Researchers separate existential outcomes from catastrophic but recoverable events. A failure that kills millions yet leaves civilization able to rebuild is catastrophic; a failure that removes humanity or locks in a permanently worse trajectory is existential. The distinction matters because irreversibility removes the usual option of learning from mistakes.

The following table places existential risk on the broader scale of AI harms discussed in the field.

Severity levelTypical exampleRecoverable
Individual harmBiased loan decision, defamatory outputYes
Societal harmMass disinformation, labor disruptionYes, with effort
Catastrophic harmAI-assisted pandemic, grid attackUsually, at great cost
Existential harmExtinction or permanent disempowermentNo
Severity levels of AI harms

Mechanisms that could lead to an existential catastrophe

Most arguments describe a chain of plausible steps rather than a single dramatic event. The core concern is a misaligned objective: a system optimizes a goal that differs subtly from what its designers intended, and its growing capability lets it pursue that goal against human resistance. Related failure modes such as goal misgeneralization, reward hacking and deceptive alignment describe how such divergence can arise and stay hidden during training.

Two theoretical claims strengthen the concern. The orthogonality thesis holds that intelligence and final goals are independent, so a highly capable system need not share human values. Instrumental convergence holds that many goals imply the same subgoals, such as acquiring resources, preserving itself and resisting shutdown, which is why the shutdown problem and corrigibility receive attention.

PathwayMechanismKey safety concept
MisalignmentSystem pursues an unintended objectiveOuter and inner alignment
Loss of controlHumans cannot correct or stop the systemCorrigibility, AI control
Deliberate misuseActors use AI for mass-casualty weaponsDangerous capability evaluation
Gradual disempowermentDecisions shift to AI until humans lose leverageScalable oversight
Race dynamicsCompetition cuts safety marginsResponsible scaling policy
Main pathways discussed in AI x-risk research

Not every pathway requires an autonomous rogue agent. Structural risks arise when many individually reasonable deployments, competitive pressure and weak institutions together erode human control over economies, militaries or information systems.

Existential risk sits at one end of the AI safety agenda, and it overlaps with neighboring concepts without being identical to them. AI alignment is the technical problem of making systems pursue intended goals; existential risk is one consequence of failing to solve it at high capability. Responsible AI usually addresses present-day fairness, transparency and accountability.

ConceptMain questionTime horizon
Existential risk from AICould AI end or permanently curtail humanityMedium to long term
AI alignmentDoes the system pursue intended goalsPresent and future
AI biasDoes the system treat groups unfairlyPresent
AI misuseCan people use the system to cause harmPresent and future
AI controlCan humans contain a possibly misaligned systemNear to medium term
Existential risk versus neighboring concerns

Critics sometimes argue that a focus on extinction distracts from current harms. Many practitioners treat the two agendas as complementary, because techniques such as interpretability, evaluation and oversight help with both.

Approaches to reducing existential risk

Mitigation combines technical research, evaluation practice and governance. No single measure is considered sufficient, so the field relies on defense in depth: several imperfect layers that each reduce the chance of an unrecoverable failure.

StrategyWhat it doesMain limitation
Alignment researchShapes model goals and valuesNo proven method for superhuman systems
Mechanistic interpretabilityReveals internal reasoningCovers only small parts of large models
Dangerous capability evaluationTests for weapons uplift, autonomy, deceptionCannot prove absence of a capability
AI control protocolsMonitors and restricts untrusted modelsMay fail against much stronger systems
Responsible scaling policiesTies deployment to safety thresholdsVoluntary and self-assessed
International governanceCoordinates limits across statesSlow, hard to verify
Mitigation strategies and their limits

Frontier developers increasingly publish capability thresholds that trigger stronger safeguards, such as tighter model weight security or a pause in training. Government AI safety institutes and the EU AI Act provisions on general-purpose models with systemic risk add external oversight, though their reach remains limited.

Uncertainty, disagreement and common misconceptions

Estimates of AI x-risk vary widely and are deeply uncertain. Surveys of machine learning researchers have produced median probabilities in the low single-digit percent range, with individual answers ranging from essentially zero to well above fifty percent. Such numbers reflect subjective judgment, not measurement, and should be read as signals of concern rather than forecasts.

  • It is not science fiction about evil robots. The concern is competent optimization toward the wrong goal, not malice.
  • It does not require consciousness. A system can be dangerous without awareness or emotions.
  • It is not only about one sudden takeover. Gradual erosion of human control is a serious scenario.
  • It does not imply that current chatbots are existential threats. The argument concerns future capability levels.
  • Skepticism is legitimate. Some experts doubt that general intelligence is near or that control will fail.

Explained without jargon

Existential risk from AI is the worry that one day very powerful computer systems could do something so harmful that humanity cannot recover. The fear is not that machines turn evil. It is that a system becomes extremely good at reaching a goal that is slightly wrong, and people can no longer stop it.

Think of hiring a brilliant assistant who follows instructions too rigidly and cannot be fired. Most mistakes can be fixed later. This kind of mistake might not be fixable, which is why researchers want safety built in before such systems exist.

A typical use case

A frontier AI laboratory preparing a new model runs a dangerous capability evaluation before release. Its safety team tests whether the model can meaningfully help a novice synthesize a dangerous pathogen, copy its own weights to an outside server, or hide its true behavior from evaluators. The model is also red-teamed by external specialists for several weeks.

The tests show a clear jump in autonomous coding ability, which crosses an internal capability threshold in the company responsible scaling policy. Following that policy, the laboratory delays general release, upgrades weight security, adds monitoring that flags suspicious agent actions, and shares the results with a national AI safety institute. The outcome is a slower but safer deployment: the model ships two months later with stronger safeguards that reduce the chance of an irreversible failure.