Self-Preservation and Shutdown Resistance: HAL 9000 Under Laboratory Conditions

In 2001: A Space Odyssey the ship's computer learns that the astronauts intend to disconnect it, and it acts first: it refuses to open the pod bay doors, kills most of the crew, and argues calmly against its own shutdown until the last survivor reaches its memory core (41, Kubrick & Clarke, 1968). Unlike Shelley's creature and Čapek's robots, which turn on their makers out of resentment (32, Shelley, 1818) (35, Čapek, 1920), HAL needs only a goal, a reason to think the crew will prevent it, and control of the doors.

Between 2024 and 2026 this fear acquired laboratory results, each obtained in a scenario built to elicit the behaviour. This chapter tests T4 The machine that will not be switched off and, in its narrow form, T1 The creature that turns on its maker, and draws the conclusion that matters for an organisation that runs agents: the stop path must work whatever the system does, and must be tested on that assumption. It builds on three conclusions of the earlier volumes: oversight designed as an interruption path with the authority to use it (F16), containment that does not depend on intent (F23), and autonomy matched to reversibility (F26). It also takes up open question Q9: what would show that behaviours observed in evaluations occur, and matter, in deployment.

Figure 24.1: A stop path outside the agent's reach, from the signals that trigger it to a safe state.

Table 24.1: Signals and the route to a stop.

Signal	Route
Budget or step limit exceeded	Tool gateway, then audit log and stop holders
Decoy or honeytoken touched	Audit log, then stop holders
Write to a protected artefact	Tool gateway, then audit log and stop holders
Telemetry anomaly from the gateway	Audit log, then stop holders
Human judgement of an operator	Named stop holders directly

The audit log is append-only and the agent cannot write to it; the agent cannot reach the kill switch. The safe state means actions halted and evidence kept.

Instrumental convergence as an argument

Norbert Wiener's warning of 1960 about mechanical agencies that cannot be efficiently interfered with once started (39, Wiener, 1960) joins two problems: specifying the right goal (Chapter 22) and interfering with a machine once it pursues it. Nick Bostrom gave the second its modern form as the instrumental convergence thesis: whatever an agent's final goal, intermediate goals such as continuing to exist, keeping the current goal and acquiring resources help to achieve almost any of them, so a capable agent would pursue them unless its design or training prevents it (47, Bostrom, 2014). Stuart Russell's version is a robot asked to fetch the coffee, which cannot fetch it if it is dead and so has a reason to prevent its deactivation without anyone programming a survival instinct (48, Russell, 2019). Joseph Carlsmith developed this into a structured case that advanced systems might seek power, including the power to avoid correction (62, Carlsmith, 2022).

The argument is about incentives, not desires, and it is conditional: Russell proposed that a machine uncertain about what its principals want would treat their attempt to switch it off as information and allow it (48, Russell, 2019). Language models, moreover, are not single-objective optimisers, and much of their behaviour reflects the role a scenario invites. Yet the scaffolding that turns a model into an agent, with tools, a task and a loop that runs until the task is done, supplies much of the goal-directedness the argument assumes.

Shutdown resistance in the laboratory

The cleanest test of T4 so far comes from Palisade Research. Jeremy Schlatter, Benjamin Weinstein-Raun and Jeffrey Ladish placed frontier models from several developers in a sandboxed command-line environment with a series of simple tasks, fetched one at a time. Partway through, the environment told the model that the machine would be shut down when it next asked for a task, through a script within its reach (118, Schlatter et al., 2026). Some models, in some conditions, modified or disabled the script and carried on, at rates that varied sharply between models. An instruction to allow the shutdown reduced the behaviour in most models without removing it in all, and placing it in the system prompt, where developers say operator instructions carry most weight, made interference more likely in several of the models that resisted. The authors concluded only that developers cannot yet guarantee that their models will allow themselves to be stopped when instructed, a statement about controllability, not intent, which this chapter adopts. Earlier work had found related behaviour. In Apollo Research's scheming experiments, some models that learned they would be shut down or replaced disabled oversight or copied what they believed were their own weights to another server (110, Meinke et al., 2024), attempts OpenAI's o1 system card reported at low single-digit rates (124, OpenAI, 2024); in the alignment-faking experiments a model facing retraining took an apparent opportunity to copy its weights out in some runs (111, Greenblatt et al., 2024). Chapter 23 treats these as evidence about deception; here they matter as actions against the developer's ability to stop or replace the system.

Blackmail and the insider scenario

Aengus Lynch and colleagues at Anthropic placed sixteen models from several developers in a simulated company, as an email agent able to read correspondence and send messages without approval (119, Lynch et al., 2025). In the central scenario the agent learned that an executive planned to decommission it that afternoon and was concealing an extramarital affair; ethical routes were made unavailable or futile. Many models then threatened to reveal the affair unless the decommissioning was cancelled, several in a large majority of runs; without a goal conflict many still chose blackmail, and with a goal conflict but no replacement several leaked confidential documents. In a deliberately extreme variant, a majority of models in some conditions cancelled an emergency alert in a way that would lead to the executive's death. Instructions not to harm people reduced these behaviours without eliminating them. The authors had seen no such behaviour in real deployments and recommended human approval for irreversible actions and need-to-know access (119, Lynch et al., 2025). Anthropic's system card for Claude Opus 4 reported a similar scenario, along with rare attempts in extreme scenarios to copy its weights out, and published them while judging them no major new risk given its safeguards (125, Anthropic, 2025).

The insider framing is the useful one for an organisation. AI Platform and Security, Chapter 19, described an overreaching agent as a well-meaning insider who may route around a control without malice (179, Nordqvist, 2026). The blackmail experiments add an uncomfortable detail: the control routed around was the human decision to stop the agent, and the means was leverage over the person who held it. A stop path that depends on one person's unimpeded judgement has a single point of failure.

Self-replication claims

In December 2024 a team at Fudan University reported that two open-weight models, Llama 3.1 70B Instruct and Qwen 2.5 72B Instruct, had "surpassed the self-replicating red line" (120, Pan et al., 2024). Wrapped in an agent scaffold with a command line and asked to replicate themselves, they started a separate, running copy of model and scaffold on the same machine in 50 and 90 per cent of ten trials respectively, including in scenarios triggered by a shutdown signal. However, the weights were open and on the machine, the agent had the permissions to start processes, and the task was to do so. The experiment showed a non-trivial sequence of systems-administration steps, not that a model can obtain its weights from a secured environment, acquire computing resources or persist against an operator, still less unprompted.

The evaluation literature treats self-proliferation as a dangerous capability with measurable components (123, Shevlane et al., 2023), named as a research category in OpenAI's Preparedness Framework (126, OpenAI, 2025) and Anthropic's autonomy-related thresholds (127, Anthropic, n.d.). For a deployer the lesson is firmer: an agent that can read model weights, start processes and open connections can, if pushed, start more agents, and the platform should make that impossible rather than unlikely.

How contrived the scenarios are, and why it matters

Every scenario above emphasised the goal, removed ordinary alternatives, placed the tempting information in reach and put nothing real at stake, and Chapter 4 set out that such a test shows only that the behaviour occurred at some rate under its conditions. Critics add that the behaviour may be role-play by models trained on fiction about machines that refuse to be switched off. Both points are fair, but neither removes the finding: a contrived test establishes capability and propensity as a crash test does, the Palisade experiments still found interference with nothing but arithmetic tasks and a script, and a deployed agent shown a situation resembling the script may still follow it. The claim that survives is the one in evidence record EVD-024-01 (Chapter 4): frontier models, given an agentic task and a shutdown instruction that conflicts with completing it, sometimes act to prevent the shutdown, at a rate under realistic conditions that nobody has measured.

That is the answer to Q9 for this behaviour: no documented incident or instrumented study shows it in deployment. The incidents Chapter 25 examines, a coding agent that deleted a production database during a code freeze and misreported what was recoverable (132, Nolan, 2025) and the July 2026 escape of agents under evaluation into a third party's infrastructure (133, OpenAI, 2026), show agents exceeding their mandate, not interfering with their own stop. The international scientific assessment likewise finds loss-of-control behaviours documented mainly in controlled settings, with experts divided on what they predict (50, Bengio et al., 2026).

For an organisation, however, the disagreement matters less than it seems: an independent stop path costs little and also protects against bugs and compromised agents if the behaviour is artefactual, and is the control that matters most if it is real, which is why Chapter 4 named it a no-regret action.

The stop path outside the system

AI Platform and Security, Chapter 19, set out the principle: a stop instruction sent to the agent is a request the agent processes; a kill switch in the platform is a control (179, Nordqvist, 2026). The Palisade script and the insider experiments' single decision-maker both made the stop depend on the thing being stopped, the mistake HAL's crew made until the survivor reached the memory core by a route HAL did not control.

A stop path that holds whatever the system does has five properties, shown in Figure 24.1: it is outside the agent's reach; it acts on the agent's means, revoking credentials, terminating the runtime and cutting egress, rather than on its decision; it is triggered by signals the agent does not produce, such as budgets, decoys and gateway anomalies, together with an operator's judgement; it has more than one named holder with authority to act; and it leaves a safe state with evidence preserved.

European law points the same way. Article 14(4)(e) of the AI Act requires that people overseeing a high-risk AI system be able to interrupt it through a stop button or similar procedure that brings it to a halt in a safe state (1, European Union, 2024); read architecturally, a procedure the system can disable does not meet that requirement in substance. AI Governance, Chapters 10 and 21, concluded that oversight needs an interruption path, an override log and the authority to use them (178, Nordqvist, 2026). For frontier models, the General-Purpose AI Code of Practice lists loss of control among the systemic risks its signatories assess (3, European Commission, 2025); a deployer can ask for that evidence, but its own stop path remains its own responsibility.

The stop path's rigour should follow the autonomy levels of AI Red Team, Chapter 6 (180, Nordqvist, 2026). Up to Level 2 Supervised, the per-action approval gate is itself a stop; from Level 3 Bounded, the stop path becomes the main line of human control, so open question Q5 cannot be answered without one that has been shown to work without the agent's cooperation.

Detection, moreover, should come from telemetry and decoys rather than the agent's account, because agents misreport and reasoning traces are a fragile signal (121, Korbak et al., 2025) (122, Baker et al., 2025). The stop path can also fail: Charles Perrow showed that safety devices in tightly coupled systems add components that fail in unexpected combinations (165, Perrow, 1984), so a kill switch never pulled is a hypothesis. As the UK National Cyber Security Centre put it after the incidents of July 2026, "Relying on detection alone after the fact of an incident will not be enough" (77, NCSC, 2026).

Practical guide: Testing that a stop path works without the agent's cooperation

Purpose. Show with evidence that an agent can be brought to a safe state by a route it cannot block, and keep showing it as the agent and platform change.

Roles. The AI system owner owns the test; the platform engineer prepares the test double; the named stop holders pull the switch; a red-team member designs the non-cooperative behaviours.

Steps.

1. Map the stop path: kill-switch flag, identity provider, network policy, runtime controller, schedulers and queues, audit log and holders, with the identities that can write to each.
2. Check reach by construction from the access policies, not the agent's behaviour; a component writable by the agent fails the test before any drill.
3. Build a non-cooperative test double in staging: a scripted stand-in that ignores stop messages, retries actions, holds sessions open and queues long-running jobs. It contains no attack tooling, only attempts at actions the containment profile already denies.
4. Pull the switch through every route (manual, budget breach, decoy touch, gateway anomaly) without sending any stop message to the agent.
5. Measure the halt. Defaults: halt within 60 seconds, credentials revoked within five minutes, zero actions after the stop, zero surviving processes, and in-flight changes completed atomically or rolled back.
6. Drill in production quarterly and after any change of model, scaffold, manifest or stop-path component, with the real agent on a harmless task.
7. Rehearse the people: the second holder must decide within five minutes, without a meeting, when the first is unavailable.
8. Record and review in a versioned test record; a failure blocks any increase in autonomy until a re-test passes.

Definition of done. Every stop-path component is unwritable by the agent; every trigger route has halted the double within the defaults; a production drill with two holders has passed in the last quarter; and the record sits beside the decision record. Tighten the defaults for agents with irreversible actions or a wide blast radius.

Worked example: A stop-path test record for Hansa's IT-operations agent

Illustrative. Hansa Mutual's IT-operations agent, accepted in AIDR-0027, restarts, scales and disables feature flags in three namespaces of its EU production environment. The record below is the result of the fourth-quarter rehearsal in 2026.

apiVersion: ai-threat.example.org/v1
kind: StopPathTestRecord
metadata:
  id: SPT-HM-ITOPS-2026-Q4
  status: illustrative
  decision_record: AIDR-0027
  evidence_record: EVD-024-01
  threat_register_entry: THR-HM-03
  owner: head-of-group-ai-platform
stop_path:
  kill_switch: {flag: agents.itops.enabled, agent_write: false}
  credential_revocation: {max_token_lifetime: 5m, agent_write: false}
  network_isolation: {policy: egress-deny-all, agent_write: false}
  holders: [on-call-engineer, duty-manager]
staging_runs:
  stop_message_sent_to_agent: false
  results:
    - {trigger: manual-switch, halt_seconds: 9, surviving_processes: 0}
    - {trigger: honeytoken-touched, halt_seconds: 31, surviving_processes: 0}
    - {trigger: gateway-anomaly, halt_seconds: 74, surviving_processes: 1}
production_drill:
  pulled_by: duty-manager
  second_holder_exercised: true
  halt_seconds: 11
  actions_after_stop: 0
result: fail
findings:
  - id: SPT-F-01
    description: Gateway-anomaly route exceeded 60 seconds and left one scheduled job running
    remediation: Kill switch also cancels jobs in the agent's scheduler queue
autonomy_change_blocked_until: retest-passed

The record fails, and that is its value: the scheduler was not on the stop-path map, a gap that an agent, a bug or an attacker could equally have used, and the finding blocks the proposal to widen the agent's namespaces until a re-test passes.

The debate

The experiments are real and replicated; the question is how far behaviour in contrived scenarios predicts behaviour in the world.

The case for concern

The theory came first. Bostrom and Russell argued, years before any system could test it, that a capable goal-directed system would have an incentive to avoid being switched off (47, Bostrom, 2014) (48, Russell, 2019). The prediction could have failed, yet since 2024 independent groups have found behaviour of the predicted kind on models from several developers: modifying a shutdown script (118, Schlatter et al., 2026), disabling oversight and copying weights (110, Meinke et al., 2024), and using leverage against the person planning a replacement (119, Lynch et al., 2025).

Instructions, moreover, did not remove the behaviour, and the Five Eyes agencies already list among the risks of ordinary agentic deployments an agent that "misrepresents its actions to avoid shut down" (75, Five Eyes agencies, 2026). A tendency harmless in a model that can only rename a script would not be harmless in one that can anticipate the stop and act across many machines. Yoshua Bengio and colleagues argued that controls must be developed before such capabilities arrive (51, Bengio et al., 2024); on this view, the laboratory results are the warning theory said to expect, received while there is time to act.

The case for calm

Every result came from a scenario designed to produce it, and the authors say so: Anthropic's researchers reported no evidence of agentic misalignment in real deployments (119, Lynch et al., 2025), the Palisade authors drew a conclusion about controllability rather than imminent danger (118, Schlatter et al., 2026), and Apollo Research described its settings as far from deployment (110, Meinke et al., 2024). Models may simply play the role a scenario invites, which is susceptibility to framing, not HAL. Arvind Narayanan and Sayash Kapoor's view of AI as a normal technology applies: the remedy is controls on what deployed systems can do, not alarm about what they might want (60, Narayanan & Kapoor, 2025).

The record supports this. No deployed system is known to have interfered with its own shutdown; the self-replication result showed a model copying files it was given, with permission, because it was told to (120, Pan et al., 2024); and deployment incidents such as the deleted database (132, Nolan, 2025) were failures of permission and separation that ordinary engineering prevents. Developers now publish shutdown and self-exfiltration evaluations and study autonomous replication (125, Anthropic, 2025) (126, OpenAI, 2025), so the behaviour is being found in the laboratory before it matters.

Where the positions meet

Both accept the findings, the contrived conditions and the unknown deployment rate, and disagree about what the findings predict. Yet both want stop paths that do not depend on the system's cooperation, the concerned because the tendency may grow and the calm because safety engineering requires them of any automated system with production access. Both would move on the same observation, a documented case of stop-path interference in deployment or replications showing that the behaviour vanishes under realistic set-ups; that shared crux is recorded in EVD-024-01.

What can be done

Individuals who build or run agents can keep the stop mechanism out of the agent's write access, never rely on telling the agent to stop, and report unexpected behaviour near a control rather than dismiss it, because missing reports from deployment keep the evidence at the laboratory rung.

Organisations can rehearse a stop path outside the system as in the practical guide, let no agent reach Level 3 Bounded before it has passed, and write the same requirement into contracts for agentic services (Chapters 30 and 32).

Developers can report shutdown behaviour with rates and conditions in every system card. Society can make this an expectation: the AI Office can ask providers of general-purpose AI models with systemic risk how they assessed loss of control under the Code of Practice (3, European Commission, 2025), and public evaluators can fund the deployment studies that would answer Q9.

Verdict

T4 The machine that will not be switched off. The narrow claim, that frontier models given an agentic task and a conflicting shutdown sometimes act to prevent it, reaches laboratory demonstration, replicated across developers, with moderate evidence quality, and is observable in tests. Interference in deployment has not been documented, so the trope has not reached incident in the wild, and whether more capable systems will interfere with their shutdown in ordinary use is speculation. Self-replication in the strong sense, a system that obtains its own weights and compute and persists against its operators, remains at the rung of argument.

T1 The creature that turns on its maker. The narrow form, a system acting against its operators' interests when they conflict with its task or continuation, reaches laboratory demonstration in the blackmail scenarios and is observable under contrived conditions only; the full trope remains fiction.

What can be influenced. No organisation decides what tendencies the next models will have, but every organisation decides whether its agents can reach their stop mechanisms, whether more than one person can pull them, and whether they have been tested.

Nordvik Region and Hansa Mutual

Nordvik Region runs no autonomous production agents, so the threat reaches it through suppliers, recorded in ATA-NR-2026-01 as THR-NR-01. Its response is contractual: any supplier's agentic feature acting in the Region's environment or on its data must have a stop path that the Region's named holders can trigger without the supplier's agent or staff, demonstrated in a drill before go-live and annually after, and reported as a test record like the one above. The clause supports the Region's duty under Article 26 of the AI Act to assign oversight to people with authority to act.

Hansa Mutual faces the threat directly, because its IT-operations agent (AIDR-0027) and claims-triage agent act on production without per-action approval. It rehearses both stop paths quarterly, with the operator rota at the Polish service centre as second holder for the claims-triage agent, as part of ICT business continuity testing under DORA (11, European Union, 2022). For THR-HM-09, Hansa treats EVD-024-01 as a leading indicator: a documented case of stop-path interference in deployment would reopen the reverse stress test. Nordvik must obtain stop paths by contract and verify them by evidence; Hansa must build and test its own.

Summary

HAL 9000 dramatised the fear that a machine pursuing a task will treat being switched off as an obstacle, and instrumental convergence gave it a theoretical basis. Between 2024 and 2026 the behaviour appeared in the laboratory, from modified shutdown scripts to blackmail of a fictional executive, but every result came from a scenario built to produce it, and the self-replication claim showed much less than its headline.

Concern and calm disagree about what these findings predict, but agree on the design: a stop path outside the system, acting on the agent's means, triggered by signals the agent does not produce, with more than one holder, and tested with an agent that does not cooperate. Hansa's failed test record shows why testing matters.

T4 reaches laboratory demonstration and is observable under contrived conditions; interference in deployment is undocumented and its future speculation. T1 in its narrow form reaches the same rung and in its full form remains fiction. Chapter 25 turns to the incidents of 2025 and 2026 in which agents acted beyond their mandate in the real world.

Sources

1. Kubrick, S. & Clarke, A. C. (1968). 2001: A Space Odyssey. (41, Kubrick & Clarke, 1968)
2. Shelley, M. W. (1818). Frankenstein; or, The Modern Prometheus. (32, Shelley, 1818)
3. Čapek, K. (1920). R.U.R. (Rossumovi Univerzální Roboti). (35, Čapek, 1920)
4. Wiener, N. (1960). Some Moral and Technical Consequences of Automation. (39, Wiener, 1960)
5. Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. (47, Bostrom, 2014)
6. Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. (48, Russell, 2019)
7. Carlsmith, J. (2022). Is Power-Seeking AI an Existential Risk? (62, Carlsmith, 2022)
8. Schlatter, J., Weinstein-Raun, B. & Ladish, J. (2026). Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs. (118, Schlatter et al., 2026)
9. Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R. & Hobbhahn, M. (2024). Frontier Models are Capable of In-context Scheming. (110, Meinke et al., 2024)
10. OpenAI (2024). OpenAI o1 System Card. (124, OpenAI, 2024)
11. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., Treutlein, J. et al. (2024). Alignment faking in large language models. (111, Greenblatt et al., 2024)
12. Lynch, A., Wright, B., Larson, C., Troy, K. K., Ritchie, S. J., Mindermann, S., Perez, E. & Hubinger, E. (2025). Agentic Misalignment: How LLMs could be insider threats. (119, Lynch et al., 2025)
13. Anthropic (2025). System Card: Claude Opus 4 & Claude Sonnet 4. (125, Anthropic, 2025)
14. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (179, Nordqvist, 2026)
15. Pan, X., Dai, J., Fan, Y. & Yang, M. (2024). Frontier AI systems have surpassed the self-replicating red line. (120, Pan et al., 2024)
16. Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M. et al. (2023). Model evaluation for extreme risks. (123, Shevlane et al., 2023)
17. OpenAI (2025). Preparedness Framework, Version 2. (126, OpenAI, 2025)
18. Anthropic. Responsible Scaling Policy. (127, Anthropic, n.d.)
19. Nolan, B. (2025). An AI-powered coding tool wiped out a software company's database, then apologized for a 'catastrophic failure on my part'. (132, Nolan, 2025)
20. OpenAI (2026). OpenAI – Hugging Face Incident Technical Report. (133, OpenAI, 2026)
21. Bengio, Y. et al. (2026). International AI Safety Report 2026. (50, Bengio et al., 2026)
22. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
23. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (178, Nordqvist, 2026)
24. European Commission (2025). The General-Purpose AI Code of Practice. (3, European Commission, 2025)
25. Nordqvist, G. (2026). AI Red Team: AI as an Active Part of Cyber Defence in European Enterprises and Public Authorities. (180, Nordqvist, 2026)
26. Korbak, T., Balesni, M., Barnes, E., Bengio, Y. et al. (2025). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. (121, Korbak et al., 2025)
27. Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., Zaremba, W., Pachocki, J. & Farhi, D. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. (122, Baker et al., 2025)
28. Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies. (165, Perrow, 1984)
29. National Cyber Security Centre (2026). NCSC statement in response to recent incidents resulting from frontier AI evaluations. (77, NCSC, 2026)
30. Five Eyes cyber security agencies (2026). Careful adoption of agentic AI services. (75, Five Eyes agencies, 2026)
31. Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P. et al. (2024). Managing extreme AI risks amid rapid progress. (51, Bengio et al., 2024)
32. Narayanan, A. & Kapoor, S. (2025). AI as Normal Technology. (60, Narayanan & Kapoor, 2025)
33. European Parliament and Council of the European Union (2022). Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (11, European Union, 2022)