Gradual Disempowerment and Correlated Failure

The machine dystopias of the public imagination end in a moment: the machine decides, the doors lock, the missiles launch. This chapter examines a path to the same destination that needs no moment at all: people hand decisions, attention and judgement to systems one reasonable step at a time, until the economy, the state and the organisation run on systems that optimise for something other than what people want, and nobody inside them could take the work back. Along the same path, many organisations come to depend on the same few systems, so that a single fault, executed at machine speed, fails all of them at once.

The chapter tests three tropes from Chapter 2. T8 The machine that stops, and the humans who forgot is central: Chapters 6, 14 and 15 tested it for the mind, work and the market, and this chapter tests it for whole systems. T2 The literal genie appears as many systems that each pursue a measurable proxy faithfully and faster than anyone can intervene, and T9 The pleasure machine as the comfort that makes each delegation welcome. This form of the dystopia most resembles how organisations actually adopt technology: no board decides to become dependent on systems it cannot replace; it decides, quarter by quarter, to automate a task that works. The chapter builds on three conclusions of the series: agents are governed as well-meaning insiders whose consequences are capped by boundaries (F13), oversight is designed rather than declared (F16), and speculative outlooks are answered with reversibility (F31) (178, Nordqvist, 2026).

Figure 26.1: Delegation accumulates in small, reasonable steps; reversibility tests act as checkpoints, and a failed test halts further delegation until the way back has been restored.

Table 26.1: The reversibility checkpoints.

Before the step	Test result
Ranking orders the work	If it passes, record the result and continue; if it fails, restore skill, fallback or second source before delegating further
Agent proposes, people approve	As above
Agent acts within rules	As above

Each test is drawn from a delegation register with an owner, a reversal path and a test date.

The Machine Stops, revisited

E. M. Forster wrote The Machine Stops in 1909 as a reply to the optimistic futures of his day (33, Forster, 1909). Chapter 2 summarised the plot: people living underground, served by a global Machine, and a civilisation that dies when it fails because nobody remembers how to repair it. Read with Part V behind it, the story makes three observations.

First, nobody decides to depend on the Machine; dependence is the sum of welcome conveniences, and the Machine has no goals of its own. Second, the decline is noticed and accepted: people complain, adapt, then forget, and the Mending Apparatus is itself part of the Machine, so the backup shares the fate of what it backs up. Third, the only person who sees clearly, Kuno, still practises an independent skill. Fiction is the lowest rung of the evidence ladder (Chapter 4), but the hypotheses are precise: dependence can grow without anyone choosing it, degradation can be absorbed rather than corrected, and the ability to correct depends on capacities that dependence erodes. Lisanne Bainbridge gave the third its engineering form for a single control room (164, Bainbridge, 1983), and Chapter 14 applied it to knowledge work. Stuart Russell uses Forster's story to name the risk of enfeeblement: people who can rely on machines for everything lose the ability to run their own civilisation (48, Russell, 2019). WALL-E gives the idea its comic form (43, Stanton, 2008).

Gradual disempowerment

In 2025 Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger and David Duvenaud published the most developed version of the argument (167, Kulveit et al., 2025). Two kinds of mechanism keep economies, states and cultures roughly aligned with human interests. The explicit mechanisms are human actions such as voting and consumer choice. The implicit mechanism is dependence on human participation: states need citizens to staff offices, pay taxes and fight wars, firms need workers and customers, and culture needs people to create it. Because these systems need people, they must give people some of what they want.

The authors argue that AI can weaken the implicit mechanism without anyone intending it. Each time an AI system replaces human labour or cognition, the system concerned depends a little less on people and therefore has a little less reason to remain responsive to them; a government that can administer and defend with automated systems needs less consent, so the explicit mechanisms lose force too. The argument requires neither misaligned AI, since systems that do exactly what their operators ask, in competition, are enough, nor any sudden event. The endpoint could be an effectively irreversible loss of human influence, which the authors count as an existential risk although no single step would look catastrophic. They call for measuring human influence over societal systems.

Chapter 15 introduced the argument as a concern about accountability. It is also T8 at scale, combined with T2 and T9: optimising a measurable proxy is welcome (T9), the proxy displaces the intent it stood for (T2), and the capacity to notice erodes because people no longer do the work (T8).

Decisive and accumulative risk

Atoosa Kasirzadeh has given the debate a vocabulary that this book adopts (57, Kasirzadeh, 2025). The decisive type of existential risk is an abrupt catastrophe, typically a highly capable system escaping control, as examined in Chapter 27. The accumulative type arises from many smaller AI-induced disruptions, of the kind Part III examined, that build up and interact, gradually weakening the resilience of the systems on which societies depend, until a shock that would once have been absorbed causes collapse.

The distinction matters for governance because the two types call for different evidence. A decisive risk is assessed through capability evaluations of individual systems (Chapters 22 to 25) and addressed mainly before deployment. An accumulative risk is assessed through indicators of systemic resilience, such as how much slack remains, how diverse the providers are and how many people could still do the work, and addressed continuously. Moreover, the accumulative type is the one ordinary organisations contribute to directly: very few train frontier models, but all of them decide what to delegate. The International AI Safety Report notes that reliance on a small number of general-purpose models creates single points of failure, because a fault in one model can propagate to every application that depends on it (50, Bengio et al., 2026), and Dan Hendrycks and colleagues add that competitive pressure makes delegation hard to refuse (56, Hendrycks et al., 2023).

Correlated failure at machine speed

Gradual disempowerment is slow; its counterpart is fast. When many actors delegate to systems that behave alike and act faster than people can follow, a disturbance can propagate through all of them before anyone intervenes. The best-documented example involves no modern AI.

On 6 May 2010, United States equity and futures markets fell sharply within minutes and recovered most of the loss within about half an hour. Andrei Kirilenko and colleagues analysed the E-mini S&P 500 futures records for that afternoon (166, Kirilenko et al., 2017). A large sell order was executed by an algorithm that paced its sales by trading volume; high-frequency traders bought, then sold the inventory to each other, raising the volume the seller used as its signal. They did not trigger the crash but exacerbated it, and a brief automatic pause by the exchange's safeguard helped the market recover. This is T2 at machine speed: no algorithm malfunctioned, but volume, the proxy for liquidity, failed when it came from algorithms passing the same contracts back and forth. Norbert Wiener warned in 1960 that once a machine acts faster than the people who must correct it, they had better be sure that its purpose is the one they intend, because they will not be able to interfere in time (39, Wiener, 1960). The crash also shows the remedy: automatic pauses and, afterwards, circuit breakers and price bands, mechanisms that act at machine speed but were designed by people in advance. A pre-decided limit that acts without waiting for a person is often the only form of human control fast enough.

Charles Perrow's Normal Accidents supplies the general theory (165, Perrow, 1984). Systems that combine interactive complexity, in which parts interact in ways designers did not foresee and operators cannot see in time, with tight coupling, in which a disturbance propagates before anyone can step in, will sooner or later suffer accidents no operator could have prevented. Perrow also noted that added redundancy can create new paths to failure, particularly when the backup depends on the same components as the primary.

AI deployments acquire both properties as they scale, and add correlation: when many organisations build on the same few models from the same few providers, a faulty update or an outage reaches all of them at once. This answers part of Q4, what assuming machine-speed adversaries and insiders means beyond security operations (AI Platform and Security, Chapter 25) (179, Nordqvist, 2026): any business process run by agents needs limits that trip automatically, a pause that does not depend on the system it pauses, and a way to continue at reduced capacity when the shared component fails.

Accumulative risk in one organisation

The same mechanism can be observed inside one organisation, where it is concrete enough to govern. A drafting assistant is introduced for routine letters, and staff still decide what the letters say. A ranking model orders a queue, and staff still decide each case. An agent proposes actions and staff approve them, ever faster as it proves reliable, the decay Chapter 14 described. The agent then acts within approval rules, and staff handle exceptions. Each step is reversible on paper; in practice, each also removes something that would be needed to reverse it.

Five things erode: skill, as Bainbridge predicted; process, as unused procedures go stale; capacity, as the organisation stops recruiting for work the system does; interfaces, as neighbouring systems are rebuilt around the AI system; and organisational memory, as people move on. None of this is visible in normal operation; it appears when the system is unavailable or must be replaced, and then all five appear together. That is the organisational form of Kasirzadeh's accumulative risk.

Two features hide the accumulation: no forum looks at what has been delegated in sum, and reversibility is assumed rather than tested, as when risk registers record "fallback to manual process" as a mitigation without anyone having run the manual process for years. Q6 asked how AI-proposed change can be reviewed for soundness when the reviewer depends on the AI (Architecture as Code (Chapter 27)) (177, Nordqvist, 2025); likewise, an organisation cannot judge its delegation sound if it no longer knows what undoing it would take.

Governing delegation

The answer has two parts, both in the series' discipline of governance as code: make delegation visible in one place, and test reversibility rather than assume it.

The delegation register lists every material function handed to an AI system, with the decision behind each step, the usage mode, the human role that remains and the path by which the function would be taken back. Unlike the AI inventory, which lists systems, it lists functions, because what matters is which human capacity is no longer exercised; an entry is updated whenever the delegation deepens. The reversibility test asks of one function: could the organisation perform it, at an acceptable level, without the AI system, for a defined period? A failed test does not mean removing the AI system; it means delegating no further in that function until the path back is restored. The design follows F31 (178, Nordqvist, 2026): society-wide disempowerment is speculative, loss of skill inside an organisation expected and correlated provider failure observable, and the test keeps options open for all three.

European law already asks for parts of this. Article 14 of the AI Act requires high-risk AI systems to be designed so that the people overseeing them can decide not to use the system, override its output and interrupt it (1, European Union, 2024), a right the reversibility test shows to be real. DORA requires an ICT business continuity policy (Article 11) and exit strategies for ICT services supporting critical or important functions (Article 28(8)) (11, European Union, 2022), and NIS 2 requires business continuity among risk-management measures (Article 21(2)(c)) (10, European Union, 2022). The implication is that an AI system on which a critical function depends belongs in continuity planning, with a tested way to continue without it.

Practical guide: Running a reversibility test: can we still do this without the AI?

Purpose. Establish, with evidence, whether the organisation could perform a delegated function at an acceptable level without the AI system for a defined period, and record what must be restored if it could not.

Roles. The function owner owns the test and its result; the AI system owner switches the system off or to fallback; the staff who perform the function take part, with their representatives consulted where working conditions change; the threat-register owner sets the scenario and records the result.

Steps.

1. Select the function from the delegation register, starting with those that support critical services, affect people's rights or have deepened in the last year. Default: test every such function at least once a year, and after any step that removes human review or moves more than a quarter of volume to the system.
2. Define the scenario, period and acceptable performance in advance: the event, its duration and the service levels and legal deadlines that must still be met. Default: one week for outages, one month for withdrawals.
3. Choose the test form: desk exercise first, then a live exercise of at least one working week, with a full switch to the fallback reserved for functions that passed it.
4. Protect the people affected. Keep a route back to the AI system if service would fall below the legal minimum, and name who decides to abort.
5. Run and measure throughput, backlog, errors, deadline misses and every hidden dependency, such as a fallback that calls the same provider.
6. Classify and act. Record passed, passed with conditions or failed in the delegation register. On a failure, freeze further delegation in that function until a re-test passes, restore the skill, procedure, capacity or second source, and report the gap to the forum that approved the delegation.

Definition of done. Criteria were written before the test; measurements and hidden dependencies are recorded; the result is linked to the threat register; conditions have owners and dates. Organisations with statutory deadlines or critical financial functions may need stricter defaults.

Worked example: A delegation register entry with a reversibility test and its result

The following entry is illustrative. It belongs to Nordvik Region, records the delegation of urgency ranking in the benefits process, and links to THR-NR-06 (erosion of independent judgement) and THR-NR-05 (dependence on providers) in ATA-NR-2026-01.

delegation_id: DEL-NR-BEN-01
organisation: Nordvik Region
function: Ordering incoming benefit claims by urgency
system: Benefits-triage system, claims ranker 4.0.0
usage_mode: Mode 4
decision_records: [AIDR-0007, AIDR-0011, AIDR-0018]
remaining_human_role: Caseworkers decide every case; ranking orders the queue
function_owner: Head of Benefits Unit
threat_register_links: [THR-NR-06, THR-NR-05, THR-NR-04]
reversal_path:
  procedure: Manual urgency checklist and unranked intake queue
  legal_constraints: Statutory decision deadlines and duty to give reasons
reversibility_test:
  test_id: RVT-NR-2027-01
  scenario: Claims ranker unavailable for one month
  form: Live exercise, one benefits team, four weeks, plus desk exercise for the unit
  acceptance_criteria:
    - Urgent claims receive first contact within the statutory deadline
    - No increase in decisions overturned on review
  abort_rule: Head of Benefits Unit restores ranking if urgent claims breach deadline
  result: passed_with_conditions
  findings:
    - Manual checklist referred to an application form retired in 2025
    - Two caseworkers hired since 2025 had never triaged unranked claims
    - Desk exercise showed unit-wide capacity short by about one week of work a month
  conditions:
    - action: Update manual checklist and store it with the ranker release notes
      owner: Head of Benefits Unit
      due: 2027-04-30
    - action: Add unranked triage to induction and the quarterly ranking-off day
      owner: Head of Benefits Unit
      due: 2027-06-30
  delegation_freeze: No further delegation of benefits triage until conditions are met
  next_test: 2028-02
reported_to: Audit committee

Each finding is cheap to fix now and costly in a real outage.

The debate

Because gradual disempowerment involves no single failure, both sides reason mainly from history and incentives.

The case for concern

Every step towards disempowerment is rational. Kulveit and colleagues claim not that anyone will choose to give up control, but that firms which automate will outcompete those that do not and that states which automate will find it hard to stop (167, Kulveit et al., 2025). Competitive pressure makes delegation hard to refuse even for actors who see its risks (56, Hendrycks et al., 2023), so a collective drift needs no collective decision.

Furthermore, the checks that have kept large systems responsive weaken at the same time, because principals have historically enforced their will by being needed. Kasirzadeh explains why the danger may not be noticed in time: an accumulative risk shows itself as lost resilience, not as harm, until a shock reveals that no slack is left (57, Kasirzadeh, 2025). Meanwhile the flash crash (166, Kirilenko et al., 2017) and Perrow's analysis show fast, correlated failure recurring in tightly coupled systems. On this view, slow erosion combined with fast failure is more plausible than any takeover, precisely because it requires nothing exotic.

The case for calm

Societies have delegated massively before, to markets, bureaucracies and the machines that run power grids and payment systems, and each time built institutions that kept the delegation governable: central banks, administrative courts, safety regulators, audit. Arvind Narayanan and Sayash Kapoor argue that AI is a normal technology whose effects unfold through slow diffusion, which gives institutions time to adapt, and whose control comes mainly from auditing, monitoring, liability and regulation (60, Narayanan & Kapoor, 2025).

Moreover, the record of correlated failure is also a record of correction: exchanges introduced circuit breakers after the flash crash, aviation absorbed Bainbridge's ironies through simulator training, and European law already requires oversight, continuity and exit arrangements. Finally, the argument's key premise, that AI will replace human participation broadly enough to weaken the checks on states and economies, depends on adoption that has not happened; measured effects on employment remain limited (Chapter 14). A risk that depends on decades of substitution can be watched with the indicators the argument itself proposes.

Where the positions meet

Both positions accept that delegation proceeds through small decisions whose sum no forum sees, and that pre-designed mechanisms acting at machine speed have contained correlated failure. Most importantly, both rest on reversibility. The calm side holds that societies and organisations can change course; the concern side holds that the ability to change course erodes as delegation deepens. That is a disagreement about a checkable fact, so both should accept that reversibility must be tested, not assumed. Where they still differ, on society-wide disempowerment over decades, F31 gives speculation the same answer: preserve reversibility while the evidence accumulates.

What can be done

Individuals can keep practising the skills on which their independence depends (Chapters 6 and 31) and ask how any proposed delegation at work would be undone.

Organisations can keep a delegation register, run reversibility tests and halt further delegation where a test fails. They can design circuit breakers into agent-run processes, as budgets, rate limits and automatic pauses, building on the containment of AI Platform and Security, Chapter 19 (179, Nordqvist, 2026), and diversify providers for critical functions.

Society can maintain diversity of providers through competition policy, require circuit breakers where AI runs critical services, as financial regulation already does, and develop the indicators Kulveit and colleagues call for.

Verdict

T8 The machine that stops, and the humans who forgot. That small delegations accumulate into a loss of the ability to work without the system reaches pattern at scale for automation in general, through decades of experience with control rooms and cockpits, and argument with early supporting evidence for AI; it is expected wherever reversibility is not tested. Correlated failure of shared IT services is observable at incident in the wild, and for shared AI providers expected (Chapter 15). A society that can no longer steer because it has delegated too much rests on argument; any forecast of it is speculation.

T2 The literal genie. Automated systems that each pursue a proxy and together produce an outcome nobody intended reached incident in the wild in the 2010 flash crash, without modern AI; comparable cascades of coupled AI agents are expected.

T9 The pleasure machine. That convenience leads organisations to neglect their fallbacks is observable; that it pacifies whole societies into surrendering control is speculation.

What can be influenced. Every organisation can decide what it delegates, see the sum in one register, test whether each delegation can be reversed and stop where it cannot. That is also part of the answer to Q10, how a board should treat risks that are extreme and contested: invest in reversibility.

Nordvik Region and Hansa Mutual

Nordvik Region asks whether its caseworkers could run the benefits process for a month without the claims ranker, which the quarterly ranking-off day of Chapter 14 cannot reveal. The worked example shows the result, which the audit committee received in its work on ATA-NR-2026-01; it answers the council's motion concretely, because the Region can show that it could stop, and at what cost.

Hansa Mutual asks whether claims could be handled if its main model provider failed for a week. It switched the claims assistant and the claims-triage agent in one market to the fallback models in its Frankfurt data centre for five working days. The assistant ran with lower but acceptable quality; the triage agent missed its thresholds for complex routes, so handlers' capacity became the bottleneck, linking THR-HM-06 to THR-HM-08. The test also found that one evaluation check used to approve the fallback called the primary provider's model. Hansa reconciled the result with its DORA continuity and exit arrangements and noted the accumulation risk to its policyholders (THR-HM-07). Nordvik's way back depends mostly on people and procedures; Hansa's on second sources and the people who absorb what they cannot handle.

Summary

The most plausible path to the machine dystopia may be a drift rather than a takeover, as Forster imagined. Kulveit and colleagues give the argument its modern form: as AI replaces human participation in the economy, culture and the state, the implicit check that kept those systems responsive to people weakens, without any system needing goals of its own. Kasirzadeh places this accumulative risk beside the decisive risk of the next chapter, and ordinary organisations contribute to it through what they delegate. Its fast counterpart is correlated failure, seen in the flash crash and predicted by Perrow's theory of tightly coupled systems.

Inside one organisation, small delegations erode skill, process, capacity, interfaces and memory until the documented fallback no longer exists. The response is a delegation register and reversibility tests, with further delegation frozen where a test fails. Losing the ability to work without automation is a pattern at scale for automation and expected for AI; correlated failure at machine speed has reached incident in the wild; society-wide disempowerment remains argument. Chapter 27 turns to the decisive form of the risk: the arguments that systems more capable than their makers could end human civilisation or humanity itself.

Sources

1. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (178, Nordqvist, 2026)
2. Forster, E. M. (1909). The Machine Stops. (33, Forster, 1909)
3. Bainbridge, L. (1983). Ironies of automation. (164, Bainbridge, 1983)
4. Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. (48, Russell, 2019)
5. Stanton, A. (2008). WALL-E. (43, Stanton, 2008)
6. Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D. & Duvenaud, D. (2025). Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development. (167, Kulveit et al., 2025)
7. Kasirzadeh, A. (2025). Two types of AI existential risk: decisive and accumulative. (57, Kasirzadeh, 2025)
8. Bengio, Y. et al. (2026). International AI Safety Report 2026. (50, Bengio et al., 2026)
9. Hendrycks, D., Mazeika, M. & Woodside, T. (2023). An Overview of Catastrophic AI Risks. (56, Hendrycks et al., 2023)
10. Kirilenko, A., Kyle, A. S., Samadi, M. & Tuzun, T. (2017). The Flash Crash: High-Frequency Trading in an Electronic Market. (166, Kirilenko et al., 2017)
11. Wiener, N. (1960). Some Moral and Technical Consequences of Automation. (39, Wiener, 1960)
12. Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies. (165, Perrow, 1984)
13. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (179, Nordqvist, 2026)
14. Nordqvist, G. (2025). Architecture as Code. (177, Nordqvist, 2025)
15. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
16. European Parliament and Council of the European Union (2022). Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (11, European Union, 2022)
17. European Parliament and Council of the European Union (2022). Directive (EU) 2022/2555 on measures for a high common level of cybersecurity across the Union (NIS 2 Directive). (10, European Union, 2022)
18. Narayanan, A. & Kapoor, S. (2025). AI as Normal Technology. (60, Narayanan & Kapoor, 2025)