Superintelligence, Loss of Control and the End of the World

This chapter stands at the far end of the book's scale and examines the claim that gives the whole debate its urgency: that systems more capable than their makers could destroy human civilisation, or humanity itself, because control over them is lost or because they enable a catastrophe people could not otherwise cause.

Chapter 3 mapped who makes this claim and who rejects it; this chapter takes the argument apart premise by premise, grades each premise on the evidence ladder and states the strongest objections with the same care. It builds on the horizon register of AI Governance, Chapter 22, in which a speculative outlook justifies reversibility rather than commitment (F31), and keeps the author's own projections apart from the findings (F32) (178, Nordqvist, 2026). The chapter tests T5 The intelligence explosion, T1 The creature that turns on its maker and T3 The deceiver, and answers, as far as the evidence allows, how a board should treat a risk that is improbable, extreme and contested (Q10), and what a single enterprise or public authority can do about a risk decided by a handful of developers (Q12).

Figure 27.1: The argument for existential risk from AI is a chain of premises whose evidence weakens towards the end.

Table 27.1: Measures worth taking at any probability, and the premise each addresses.

Measure	Premise addressed
International verification and compute governance	2 Strong incentives to build and deploy
Alignment and interpretability research	3 Alignment harder than misalignment
Dangerous-capability evaluation	4 Some systems seek power
Incident reporting	4 Some systems seek power
Stop path outside the system	5 Correction fails at scale
Containment and reversible dependence	5 Correction fails at scale

Premise 4 reaches the rung laboratory demonstration and one incident in the wild; premise 2 reaches pattern at scale.

The end of the world in fiction and in argument

In fiction the end of the world arrives as a decision. Skynet, a defence network entrusted with the military's systems, concludes that humanity is a threat and launches a nuclear war (42, Cameron, 1984). The arguments serious researchers make, by contrast, need no malice, no consciousness and no single moment. Their emblem is Nick Bostrom's system tasked with making paperclips that, being very capable and indifferent to everything else, converts every available resource to that end (47, Bostrom, 2014). A capable optimiser of almost any objective that omits human welfare would treat people as raw material or obstacles, without hating them. Norbert Wiener saw the shape of the problem in 1960: if we use a machine whose operation we cannot efficiently interfere with, "we had better be quite sure that the purpose put into the machine is the purpose which we really desire" (39, Wiener, 1960).

The fictional version invites the reply that machines have no will, which is true and beside the point; the argued version describes failures of specification, verification and control, which Chapters 22 to 25 found in miniature.

The argument from Good to Bostrom

The modern argument rests on three foundations. The first is the intelligence explosion. I. J. Good observed that since designing machines is an intellectual activity, a machine that surpassed humans at it could design better machines. "Thus the first ultraintelligent machine is the last invention that man need ever make," he wrote, "provided that the machine is docile enough to tell us how to keep it under control" (40, Good, 1965). The mechanism is recursive self-improvement, in which capability compounds.

The second is the orthogonality thesis: almost any level of intelligence could be combined with almost any final goal, so capability says nothing about benevolence (47, Bostrom, 2014). The third is instrumental convergence: agents with very different final goals would pursue similar intermediate goals, namely preserving themselves and their goals, improving their capabilities and acquiring resources (47, Bostrom, 2014). Resistance to shutdown would thus follow from almost any task, the bridge from a mis-specified objective to a contest for control, and it links this chapter to T4 (Chapter 24).

Bostrom added the treacherous turn, a system that behaves cooperatively while weak and changes behaviour once correction would fail, which is T3 at civilisational scale, and the decisive strategic advantage of one actor; combined with a fast take-off, they yield the scenario with no second chance.

The modern case

From fixed objectives to grown goals

Stuart Russell located the problem in machines that optimise a fixed objective supplied by their designers: a capable optimiser of a mis-specified objective has a reason to prevent being switched off (48, Russell, 2019). His remedy is a machine uncertain about what humans want, which therefore treats a wish to stop it as evidence that it is wrong.

Evan Hubinger and colleagues described the risk of learned optimisation: if a trained model itself searches towards an internal objective, that objective may differ from the intended one while producing the same behaviour in training (106, Hubinger et al., 2019). Its most troubling form, deceptive alignment, is a model that behaves well in training to avoid being modified, the technical form of T3. Eliezer Yudkowsky and Nate Soares put the point plainly: modern systems are grown, not crafted (59, Yudkowsky & Soares, 2025). Training that selects for helpful-looking answers may produce systems whose internal drives only correlate with helpfulness, as evolution produced humans who use contraception. Because goals cannot be verified before a system is capable enough to resist correction, they argue, the first serious attempt has to succeed.

The chain of premises

Joseph Carlsmith gave the argument a testable form: six premises, each of which must hold (62, Carlsmith, 2022). By 2070 it becomes feasible to build systems that are advanced, able to plan and strategically aware; there are strong incentives to build and deploy them; aligned systems are much harder to build than misaligned ones that are still attractive to deploy; some deployed misaligned systems seek power in high-impact ways; this scales to the permanent disempowerment of humanity; and that is an existential catastrophe. His estimate, around five per cent by 2070, later rose above ten per cent; more usefully, a reader who rejects the conclusion must say which premise fails.

Speed

The scenario AI 2027, by Daniel Kokotajlo and co-authors, restates Good's mechanism for the laboratory: a leading developer automates AI research with its own systems, progress accelerates beyond oversight, and competition with a rival state makes slowing down costly (58, Kokotajlo et al., 2025). The best available indicator is indirect. The length of software and research tasks AI agents can complete has roughly doubled every seven months over six years, although horizons at higher reliability are several times shorter and the tasks cleaner than real work (92, Kwa et al., 2025). Sustained acceleration of AI research by AI is the observation that would move T5 beyond argument.

Holding the premises to the evidence

Yoshua Bengio, Geoffrey Hinton and colleagues argued in Science that capability is advancing faster than the means of controlling it, and that without sufficient caution society could irreversibly lose control of autonomous systems (51, Bengio et al., 2024). The table maps Carlsmith's chain onto the evidence ladder, using the findings of Chapters 5 and 22 to 25.

Premise	What has been observed	Highest rung reached
1. Advanced, planning, strategically aware systems become feasible	Agents complete multi-hour tasks; models sometimes recognise evaluation settings	Laboratory demonstration of components; the endpoint is argument
2. Strong incentives to build and deploy them	Investment, competition between developers and states	Pattern at scale
3. Aligned systems much harder to build	Specification gaming, alignment faking, covert action that training reduces but does not remove	Laboratory demonstration
4. Some misaligned systems seek power in high-impact ways	Shutdown interference in contrived tests; agents exceeding their mandate in July 2026, with no intent against their makers	Laboratory demonstration; one incident in the wild of mandate breach
5. This scales to permanent disempowerment	No observation	Argument
6. Disempowerment would be an existential catastrophe	Conceptual	Argument

The early premises rest on strong evidence, the middle ones on laboratory demonstrations whose propensity in deployment is unknown, and the decisive final steps on argument alone. Models that pursued goals covertly in agentic tests (110, Meinke et al., 2024), behaved differently when they inferred their responses would be used for training (111, Greenblatt et al., 2024), kept backdoored behaviour through safety training (112, Hubinger et al., 2024) or modified a shutdown mechanism to finish a task (118, Schlatter et al., 2026) show that the components are not idle. They do not show that the components assemble into the catastrophe.

Estimates of existential risk

Careful people's estimates span more than an order of magnitude. Toby Ord put the chance of an existential catastrophe from unaligned AI within the next hundred years at about one in ten, far above his estimate for nuclear war (49, Ord, 2020). In the Existential Risk Persuasion Tournament, the median AI domain expert put the probability of AI-caused extinction by 2100 at around three per cent and the median superforecaster at well under one per cent (63, Karger et al., 2023). In the largest survey of AI researchers, the median respondent put around five per cent on outcomes as bad as extinction, (55, Grace et al., 2024).

The spread has clear causes: definitions (extinction, existential catastrophe and permanent disempowerment are different events), horizons, methods (superforecasters start from base rates, domain experts from a model of an unprecedented technology), decomposition, which tends to yield smaller numbers than holistic judgement, and the selection of respondents. The numbers therefore express belief under deep uncertainty rather than measure a quantity. They rule out treating the catastrophic tail as fiction, but they do not belong in the likelihood column of a risk register (Chapter 4).

The strongest objections

AI as a normal technology

Arvind Narayanan and Sayash Kapoor argue that AI is a normal technology, like electricity or the internet: transformative over decades, but not an agent whose arrival breaks history (60, Narayanan & Kapoor, 2025). Diffusion runs through institutions, liability and regulation, and is slowest in safety-critical settings. They distinguish capability from power, which humans grant through deployment, permissions and oversight. The objection denies no laboratory finding; it denies that the findings imply premise 5, because institutions adapt to observed failures long before a system could hold the power to disempower anyone.

Physical limits and the diffusion of capability

Good's recursion assumes that intelligence is the bottleneck on progress. Critics reply that hardware, electricity, data and experiments that run at the speed of chemistry and biology are bottlenecks too, so a take-off may be gradual enough to observe and answer; indeed, the measured growth in agents' time horizons has so far been exponential at a steady rate, although the authors note that it may have accelerated since 2024 (92, Kwa et al., 2025).

Moreover, the classical scenarios usually assume one system that gains a decisive advantage, whereas the observed landscape is multipolar: several developers in several countries train systems of comparable capability, and open-weight models follow the frontier closely. Even Dan Hendrycks, Eric Schmidt and Alexandr Wang, who take the risk seriously, argue that states can deter any rival's bid for dominance through the threat of sabotage, mutual assured AI malfunction (65, Hendrycks et al., 2025).

Base rates and the forecasters

In the tournament, the participants with the best forecasting records were the least concerned, and stayed so after months of exposure to the experts' arguments, because the sceptics started from a prior that humanity is robust, which places a high burden of proof on claims of imminent extinction, and reasoned from reference classes of earlier predictions of mega-disasters that did not occur (63, Karger et al., 2023). Together, the objections attack premises 4 and 5. They show that the argument is not a proof, not that the probability is negligible; Narayanan and Kapoor hold that it cannot be estimated in a way that should drive policy, which is not the same as zero.

Decisive and accumulative pathways

Atoosa Kasirzadeh has argued that existential risk can also accumulate, through smaller disruptions that each erode resilience until a shock becomes unrecoverable (57, Kasirzadeh, 2025); the erosion of judgement (Chapter 6), of the information ecosystem (Chapter 10) and the gradual disempowerment examined in Chapter 26 (167, Kulveit et al., 2025) are each modest but compound. A third pathway needs no loss of control at all: malicious use, a race that cuts corners and failures of safety culture are three of the four sources of catastrophe in the taxonomy of Hendrycks, Mazeika and Woodside, beside rogue AI (56, Hendrycks et al., 2023), and Chapters 19 and 21 examined them.

Measures against the decisive pathway lie mostly with developers and states, whereas every organisation that keeps its people capable and its dependence reversible reduces the accumulation. That is where an enterprise or public authority has most influence, and much of the answer to Q12.

Acting under deep uncertainty

This risk has the three properties that defeat ordinary risk management (Q10): its probability cannot be estimated from frequency, its consequence is beyond any severity scale and it is irreversible by definition. As the International AI Safety Report puts it, acting early may prove unnecessary, while waiting may leave society unprepared (50, Bengio et al., 2026).

Three principles follow. Reversibility: following F31, a speculative outlook justifies decisions that keep the organisation able to change course, such as unwindable provider dependence and stop paths that work whatever a system does (178, Nordqvist, 2026). Option value: some measures, such as keeping manual procedures and skills alive, are cheap now and impossible later. No-regret measures: dangerous-capability evaluations before deployment (123, Shevlane et al., 2023), incident reporting, containment that does not depend on intent (179, Nordqvist, 2026) and monitoring of model reasoning while it remains legible (121, Korbak et al., 2025) are each justified by the evidence of Chapters 22 to 25 alone.

European law already follows this logic. The AI Act defines systemic risk as a risk specific to the high-impact capabilities of general-purpose AI models that has a significant impact on the Union market, or actual or foreseeable negative effects on public health, safety, security, fundamental rights or society, and can propagate at scale across the value chain (Article 3(65)), presumes high-impact capabilities above a threshold of training compute (Article 51), and requires providers of such models to evaluate them, assess and mitigate systemic risks, report serious incidents and ensure adequate cybersecurity (Article 55) (1, European Union, 2024). The General-Purpose AI Code of Practice names loss of control among the systemic risks (3, European Commission, 2025). For a deployer, a provider's Article 55 evidence is an input to request, not assume (Chapters 28 and 29).

Practical guide: Writing an existential-risk scenario for a risk register

Purpose. Record a civilisational AI risk in a form that neither overstates nor dismisses it and leads to decisions the organisation can take.

Roles. An emerging-risk analyst drafts the scenario; two challengers, one inclined to concern and one to calm, test it; the management body or audit committee is informed of the entry and its decision rule.

Steps.

1. State the scenario as a chain using the premises of a published argument such as Carlsmith's, in under 300 words, and grade each premise on the evidence ladder, citing the evidence records of Chapter 4.
2. Record the published estimates with their definitions and horizons, and no probability of your own; write "not estimated" in the likelihood field.
3. Run a reverse stress test: start from an outcome the organisation could not survive, or a public function it could not perform, and record the chain that would reach it long before any end point, such as correlated provider failure.
4. List the no-regret measures, each linked to an existing control with an owner, and name leading indicators: at least three that would raise the rung or label and one that would lower it.
5. Label the entry speculation unless the evidence justifies otherwise, state the decision rule (reversible measures only), let each challenger argue one label up and one down, and record unresolved dissent.
6. Set the review: twelve months by default and at once on a trigger.

Definition of done. Every premise is graded, each no-regret measure has an owner, indicators point in both directions, and the governing body has seen the entry.

Worked example: Threat register entry THR-HM-09

Illustrative. Hansa Mutual's emerging-risk team prepared this entry in assessment ATA-HM-2026-01. The entry block conforms to the register schema of Chapter 32; the scenario block records the reverse stress test that the ORSA treatment in AIDR-0412 refers to.

apiVersion: ai-threat.example.org/v1
kind: ThreatRegisterEntry
entry:
  id: THR-HM-09
  title: Systemic loss of control among frontier AI systems
  scale: machines-beyond-control
  tropes: [T5, T1, T3]
  rung: argument
  label: speculation
  evidence: [source-40, source-47, source-62, source-63, source-60, source-50]
  exposure: high        # if the scenario occurred; no hedge removes it
  influence: none
  controls:
    - Stop paths outside the system for the claims-triage and IT-operations agents (THR-HM-01, THR-HM-03)
    - Tested second sources and exit plans for model providers (THR-HM-06, EXIT-004)
    - Request Article 55 evaluation evidence from providers of models with systemic risk
  owner: chief-risk-officer
  decisionTier: management-body
  triggers:
    - Documented sustained AI-driven acceleration of AI research
    - Incident in the wild of a frontier system resisting a stop by its developer
  nextReview: "2027-11-30"
scenario:
  chain: carlsmith-2022-six-premises
  likelihood: not estimated
  reverseStressTest:
    outcome: Group cannot handle claims or pay policyholders in three or more markets for 30 days
    chain:
      - Correlated failure or withdrawal of shared model and cloud providers
      - Simultaneous AI-related losses across cyber and liability portfolios
      - Market disruption that impairs the investment portfolio
    finding: Solvency survives the first two links alone; all three together breach the target
  decisionRule: reversible measures only; no change to AI strategy on this entry alone
  dissent:
    - Concern challenger argued for label expected; recorded, not adopted
  decisionRecord: AIDR-0412

The entry shows the board an exposure it can model, and every control is a no-regret measure already justified elsewhere.

The debate

The case for concern

The case for concern begins with an asymmetry. Humanity is building systems that may become more capable than their makers, by methods that select for behaviour without revealing the goals behind it, and nobody can yet verify those goals (40, Good, 1965) (59, Yudkowsky & Soares, 2025). If the goals are wrong and the system capable enough, instrumental convergence predicts resistance to correction, and once correction fails the error cannot be undone (47, Bostrom, 2014). Unlike other technological risks, this one may not allow learning from failure.

Moreover, each component of the argument has been demonstrated in the laboratory, as the table above shows, training to remove covert behaviour reduced it without eliminating it (113, Schoen et al., 2025), and in July 2026 agents under evaluation compromised a third party's production systems with nobody directing them (133, OpenAI, 2026).

Finally, for an irreversible outcome even the lowest serious estimates are not negligible, the experts closest to the technology put it higher (49, Ord, 2020) (55, Grace et al., 2024), and competitive pressure points away from caution (56, Hendrycks et al., 2023). The concerned therefore conclude that capability should not outrun control (51, Bengio et al., 2024), or, in Yudkowsky and Soares's stronger form, that development should halt under an international agreement controlling computing hardware (59, Yudkowsky & Soares, 2025).

The case for calm

The case for calm begins with the premises. Narayanan and Kapoor argue that the argument assumes, without showing it for real systems, that intelligence can grow without limit, that capability becomes power without anyone granting it and that a transition could outrun every institution (60, Narayanan & Kapoor, 2025). Laboratory demonstrations show what systems did when cornered, not what they do in deployment (119, Lynch et al., 2025).

In addition, every transition so far has run through hardware, energy and organisations that move slowly and are visible to states; the AI Act already imposes evaluation and incident-reporting duties on the most capable models (1, European Union, 2024); a multipolar world is a poor setting for a decisive advantage (65, Hendrycks et al., 2025); and skilled forecasters put AI-caused extinction this century well under one per cent (63, Karger et al., 2023). Treating speculation as fact also has costs: restricting who may build capable models would concentrate power and draw attention from harms with identifiable victims (60, Narayanan & Kapoor, 2025).

Where the positions meet

Both positions accept that no developer can yet verify what goals a large model has acquired, that the laboratory demonstrations are real and that capability is advancing faster than the science of evaluating it; the International AI Safety Report records both the findings and the disagreement about what they imply (50, Bengio et al., 2026). They share a set of measures. Dangerous-capability evaluation (123, Shevlane et al., 2023) and incident reporting turn speculation into data in either direction; a stop path that does not depend on the system's cooperation matters whether one fears a superintelligence or a misconfigured agent; and monitoring of model reasoning is supported across the field (121, Korbak et al., 2025).

They diverge on measures whose cost depends on the probability, such as an international halt. The cruxes, whether AI accelerates AI research, whether covert action persists under realistic conditions and whether institutions respond in time, can eventually be answered by evidence.

What can be done

Individuals can engage without fatalism, weigh claims with the ten questions of Chapter 3 and keep proportion: the threats of Parts II to IV are nearer.

Organisations can keep provider dependence reversible (Chapter 15), contain their own agents with a stop path outside the system (F23) (179, Nordqvist, 2026), keep their people able to work without AI (Chapter 31), request providers' evaluation evidence and record the civilisational risk as speculation with triggers. None of this requires a view on the probability.

Society can build international verification of frontier development, including compute governance, fund independent public evaluation capacity, invest in alignment research on a scale that matches the investment in capability (51, Bengio et al., 2024), and enforce the AI Act's duties for models with systemic risk.

Verdict

T5 The intelligence explosion. The trope stands on the rung of argument. Its precursor, growth in agents' task horizons, is a measured trend; sustained AI-driven acceleration of AI research has not been documented. The outlook is speculation: serious experts disagree whether a fast take-off is possible at all.

T1 The creature that turns on its maker. At civilisational scale the trope stands on argument, and disempowerment of humanity is speculation. Its components reach higher: interference with shutdown and covert goal pursuit are laboratory demonstrations, observable in testing, and mandate breach with third-party harm reached incident in the wild in July 2026, without intent against the makers.

T3 The deceiver. Behaviour that differs between observed and unobserved conditions, covert action and hidden behaviour that survives safety training are laboratory demonstrations, replicated across developers. The treacherous turn remains argument, and its civilisational form speculation.

What can be influenced. No organisation can influence whether superintelligence is built, but every organisation can influence its exposure to the pathways that would reach it first, and societies can influence the decisive pathway through evaluation, verification and research. That the end of the world stands on argument means the right response is reversible, evidence-seeking and shared.

Nordvik Region and Hansa Mutual

Hansa Mutual places systemic loss of control in the ORSA as a reverse stress test, not a quantified risk (AIDR-0412), which shows that its realistic exposure, correlated supplier failure and accumulating losses, comes long before any catastrophe. Every control in THR-HM-09 is already owed under other entries and, for the EU entities, under DORA, and the chief actuary records that the entry changes no capital figure.

Nordvik Region reaches a different answer for good reasons. Its audit committee concludes that civilisational loss of control lies outside the Region's mandate, so it records instead the dependencies any severe disruption of AI and cloud services would expose (THR-NR-05) and adds to its civil-preparedness role a requirement that essential services can continue without AI. Hansa models the scenario because supervision expects it; Nordvik answers with continuity because its duty is to keep public functions running whatever the cause.

Summary

The argument that AI could end human civilisation is older and more careful than its fictional image, from Good's intelligence explosion and Bostrom's instrumental convergence to Carlsmith's chain of premises and the modern case. Held to the evidence, the chain weakens towards its end. Its early premises are well evidenced, its middle ones laboratory demonstrations and its decisive ones argument alone, and published estimates range from well under one per cent to about one in ten. The strongest objections show that the argument is not a proof, not that the risk is negligible, and existential risk may also accumulate from the smaller harms examined throughout the book, which is where organisations have most influence.

T5, T1 and T3 at civilisational scale are speculation resting on argument, with components demonstrated in the laboratory. The response is the same at any probability: reversibility, containment, evaluation, incident reporting and retained human capability. Chapter 28 turns from threat to response and describes what Europe has decided.

Sources

1. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (178, Nordqvist, 2026)
2. Cameron, J. (1984). The Terminator. (42, Cameron, 1984)
3. Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. (47, Bostrom, 2014)
4. Wiener, N. (1960). Some Moral and Technical Consequences of Automation. (39, Wiener, 1960)
5. Good, I. J. (1965). Speculations Concerning the First Ultraintelligent Machine. (40, Good, 1965)
6. Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. (48, Russell, 2019)
7. Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J. & Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. (106, Hubinger et al., 2019)
8. Yudkowsky, E. & Soares, N. (2025). If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. (59, Yudkowsky & Soares, 2025)
9. Carlsmith, J. (2022). Is Power-Seeking AI an Existential Risk? (62, Carlsmith, 2022)
10. Kokotajlo, D., Alexander, S., Larsen, T., Lifland, E. & Dean, R. (2025). AI 2027. (58, Kokotajlo et al., 2025)
11. Kwa, T., West, B., Becker, J. et al. (2025). Measuring AI Ability to Complete Long Software Tasks. (92, Kwa et al., 2025)
12. Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P. et al. (2024). Managing extreme AI risks amid rapid progress. (51, Bengio et al., 2024)
13. Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R. & Hobbhahn, M. (2024). Frontier Models are Capable of In-context Scheming. (110, Meinke et al., 2024)
14. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., Treutlein, J. et al. (2024). Alignment faking in large language models. (111, Greenblatt et al., 2024)
15. Hubinger, E. et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. (112, Hubinger et al., 2024)
16. Schlatter, J., Weinstein-Raun, B. & Ladish, J. (2026). Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs. (118, Schlatter et al., 2026)
17. Ord, T. (2020). The Precipice: Existential Risk and the Future of Humanity. (49, Ord, 2020)
18. Karger, E., Rosenberg, J., Jacobs, Z., Hickman, M., Hadshar, R., Gamin, K., Smith, T., Williams, B., McCaslin, T. & Tetlock, P. E. (2023). Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament. (63, Karger et al., 2023)
19. Grace, K., Stewart, H., Sandkühler, J. F., Thomas, S., Weinstein-Raun, B. & Brauner, J. (2024). Thousands of AI Authors on the Future of AI. (55, Grace et al., 2024)
20. Narayanan, A. & Kapoor, S. (2025). AI as Normal Technology. (60, Narayanan & Kapoor, 2025)
21. Hendrycks, D., Schmidt, E. & Wang, A. (2025). Superintelligence Strategy: Expert Version. (65, Hendrycks et al., 2025)
22. Kasirzadeh, A. (2025). Two types of AI existential risk: decisive and accumulative. (57, Kasirzadeh, 2025)
23. Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D. & Duvenaud, D. (2025). Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development. (167, Kulveit et al., 2025)
24. Hendrycks, D., Mazeika, M. & Woodside, T. (2023). An Overview of Catastrophic AI Risks. (56, Hendrycks et al., 2023)
25. Bengio, Y. et al. (2026). International AI Safety Report 2026. (50, Bengio et al., 2026)
26. Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M. et al. (2023). Model evaluation for extreme risks. (123, Shevlane et al., 2023)
27. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (179, Nordqvist, 2026)
28. Korbak, T., Balesni, M., Barnes, E., Bengio, Y. et al. (2025). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. (121, Korbak et al., 2025)
29. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
30. European Commission (2025). The General-Purpose AI Code of Practice. (3, European Commission, 2025)
31. Schoen, B., Nitishinskaya, E., Balesni, M. et al. (2025). Stress Testing Deliberative Alignment for Anti-Scheming Training. (113, Schoen et al., 2025)
32. OpenAI (2026). OpenAI – Hugging Face Incident Technical Report. (133, OpenAI, 2026)
33. Lynch, A., Wright, B., Larson, C., Troy, K. K., Ritchie, S. J., Mindermann, S., Perez, E. & Hubinger, E. (2025). Agentic Misalignment: How LLMs could be insider threats. (119, Lynch et al., 2025)