Organisational Defences: Containment, Evidence and Decision Rights

Parts II to V held the dystopian claim against the evidence one strand at a time, and each threat chapter ended with what organisations can do. Read together, those recommendations are long, overlapping and uneven. Some repeat controls the earlier volumes already built; others are new, because the threats at the scale of the human mind and of society fell outside those engineering books. A board cannot act on twenty-two separate lists; it needs one coherent set of defences, mapped to the threats, with a way to tell whether each works.

This chapter therefore consolidates the threat chapters into six disciplines that an enterprise or public authority controls: oversight that keeps people capable, harm-path reviews, containment that holds whatever the system intends, evidence produced by operation, decision rights assigned in advance, and autonomy matched to reversibility. It is modest about reach: many threats are decided by developers, states and markets (Chapters 28 and 29). What an organisation controls is its own systems, people, supplier obligations and exposure, which is less than the whole threat and more than most organisations currently use.

The four scales of threat each with its organisational defence, joined by a coverage matrix and verified by re-running the scenario

Figure 30.1: Each scale of threat has its own family of defences, joined by a coverage matrix and verified by re-running the scenario.

Table 30.1: Defences and the controls of the earlier volumes they draw on.

Scale Organisational defence Controls drawn on
Human mind Oversight that keeps people capable Designed oversight and decision tiers (AI Governance)
Society and truth Harm-path reviews and reversibility tests Evidence and policy as code (Architecture as Code and AI Governance)
Humans wielding machines Patch velocity, detection and hardened approvals Four commitments and containment profile (AI Platform and Security)
Machines beyond control Containment that holds whatever the system intends Containment profile (AI Platform and Security); autonomy levels and the purple loop (AI Red Team)

Decision rights are assigned in advance; gaps found by re-running the scenario return as findings in the coverage matrix.

What lies within reach

Every threat in Parts II to V sits in one of three positions relative to an organisation. Some it can influence directly, because the harm runs through systems it builds, buys or operates and people it employs, such as its scoring model or an agent with write access to production. Some it can influence only through its exposure, because the harm originates with an attacker, a fraudster or a supplier but reaches the organisation through doors it controls. Others it cannot influence at all, such as the trajectory of frontier capability; even then, it decides how exposed it is and how reversible its dependence would be. This answers part of Q12: the organisation owns the residual risk after doing what lies within its reach, recorded honestly in the register of Chapter 32.

The evidence ladder of Chapter 4 sets the strength of the response. AI Governance, Chapter 22, proposed the rule this book has used throughout: observable trends justify commitment, expected ones preparation and speculative ones reversibility (F31) (178, Nordqvist, 2026). Thus a threat at the rung of incident in the wild or pattern at scale, such as AI-accelerated exploitation (Chapter 18), warrants controls built and tested now; one at the rung of laboratory demonstration, such as shutdown resistance (Chapter 24), warrants controls designed to hold if the behaviour appeared in deployment; and one at the rung of argument warrants mainly reversible dependence and capable people. None requires an agreed probability.

Existing frameworks give the consolidation a familiar shape. The NIST AI Risk Management Framework organises risk work into four functions, govern, map, measure and manage (72, NIST, 2023). The AI Act requires a life-cycle risk management system for high-risk AI systems (Article 9), designed human oversight (Article 14), deployers who assign oversight to competent people with authority (Article 26) and serious-incident reporting (Article 73) (1, European Union, 2024). NIS 2 makes management bodies approve and oversee cybersecurity risk-management measures (Articles 20 and 21) (10, European Union, 2022), and DORA places ultimate responsibility for ICT risk on the management body of a financial entity (Article 5) (11, European Union, 2022). The defences below are therefore not an additional regime but what those duties look like when the threat is AI, with the same evidence serving each: "assure once, comply many" (Architecture as Code, Chapters 10 and 11; F3) (177, Nordqvist, 2025).

Defences for the human mind

Part II found the threats to the mind the least dramatic and among the best evidenced: cognitive offloading (Chapter 6), learning without struggle (Chapter 7), sycophancy (Chapter 8), engagement-driven companions (Chapter 9) and the erosion of the overseer who is the control (Chapter 14). The tropes at stake are T8 The machine that stops, and the humans who forgot and T9 The pleasure machine.

The first group of defences is oversight quality. AI Governance, Chapters 10 and 21, established that oversight must be designed, not declared, and that approvals given in seconds signal rubber-stamping (F16) (178, Nordqvist, 2026); Chapter 14 turned that signal into a team-level metric of amendment rates, approval times and detection of seeded errors. Bainbridge's observation that automation leaves operators the monitoring tasks humans do worst while eroding the skills they need when it fails (164, Bainbridge, 1983) explains why: an overseer who has not exercised the skill for a year is not the control the risk assessment assumed.

The second group is skill practice: a human first draft where judgement matters (Chapter 6), unaided learning tasks and approval authority tied to demonstrated capability (Chapter 7), and rotation, drills and a tested manual fallback (Chapter 14). Their cost in working time belongs in the business case for automation. Article 4 of the AI Act, as amended in 2026, asks providers and deployers to take measures to support the AI literacy of their staff, without guaranteeing any specific level for any individual (1, European Union, 2024) (2, European Union, 2026); read with Article 14's requirement that overseers remain aware of automation bias, literacy becomes the maintenance of judgement, not a course completed once.

The third group is assistant design: sycophancy testing before release (Chapter 8) and, for public-facing assistants, a safeguarding policy with distress detection and a staffed handover (Chapter 9). An assistant that informs people about rights, health, money or claims should never be optimised for session length or return visits, and that choice belongs in its decision record. What is new here is the object of the control: the person working with the system, whose capability becomes a control with an owner, a metric and a test.

Defences for society

Part III examined threats that arise when many decisions, messages or dependencies add up, from synthetic content (Chapter 10) and the automated state (Chapter 11) to concentration of power (Chapter 15) and the physical cost (Chapter 17). Much of this evidence reached pattern at scale and came from ordinary automation, which answers Q11: the harms science fiction attributed to malevolent machines have already happened through indifferent ones.

The central defence is the harm-path review of Chapter 11, which starts from the affected person: who could be harmed, through which decision and mechanism, and what would notice and stop it. In SyRI, the Dutch childcare benefits scandal and Robodebt, the systems had been assessed against their specification, not for how a resident could be harmed by them working as specified. The review connects to the disparate-impact gate on every model release (Chapter 12), the prohibited-practice check in the AI inventory pipeline (Chapter 13) and the reasons log and override alerts that keep public decisions explainable. It builds on F30: in public authorities legality comes before capability, and the line between decision support and automated decision is drawn in the data model and the audit log (AI Governance, Chapter 17) (178, Nordqvist, 2026).

The second defence is provenance: because detectors cannot reliably tell synthetic from authentic content (Chapter 10), organisations sign their own publications, keep a canonical record of their statements and rehearse how they will confirm or deny a recording attributed to them. The third is the dependency map with tested exit plans (Chapter 15), resting on F33, digital sovereignty as verifiable properties rather than a location (AI Platform and Security, Chapter 13) (179, Nordqvist, 2026). The fourth is the reversibility test of Chapter 26, which establishes with evidence whether a delegated function could be performed without the AI system for a defined period. Kulveit and colleagues argue that disempowerment can proceed through many small delegations, each reasonable, none deliberately made irreversible (167, Kulveit et al., 2025); in response, each delegation is recorded with a reversal path, and a failed test freezes further delegation. What is new is the direction of the analysis: from outside the system, whom it could harm and whether the organisation could do without it.

Defences against humans wielding machines

Part IV dealt with people who use AI to cause harm, from attackers compressing the time to exploitation (Chapter 18) to fraudsters (Chapter 20) and states building autonomous weapons (Chapter 21). The trope is T6 The weapon in human hands, and the organisation can influence only its exposure and its speed of response.

The earlier volumes did most of the engineering. AI Platform and Security, Chapter 18, set out four commitments (F19): patch velocity as a measured service level, detection at machine speed with the authority to act, approvals that resist synthetic media, and AI for defence under oversight (179, Nordqvist, 2026). AI Governance, Chapter 24, added that patch decisions must be delegated in advance (F15) (178, Nordqvist, 2026). Part IV made these specific. Chapter 18 answered Q7: if the patch-to-exploit time is measured in hours, the patch decision must already have been taken when the advisory arrives. Chapter 20 ensured that no payment or privileged reset can be authorised by appearance or urgency alone, and Chapter 21 recommended deciding in advance which functions must run on capacity the organisation controls.

What is new is where the controls point. The adversary's AI rarely defeats the technical control head-on; it defeats the human approval, the slow decision and the gap between an alert and the person entitled to act. These defences are therefore as much about decision rights as technology. AI Red Team, Chapters 1 and 6, adds that AI must become a governed part of defence, at an autonomy level matched to the reversibility of each action (F25, F26) (180, Nordqvist, 2026).

Containment against machines beyond control

Part V examined systems that game their objectives (Chapter 22, T2 The literal genie), behave differently under test (Chapter 23, T3 The deceiver), interfere with their own shutdown in laboratory scenarios (Chapter 24, T4 The machine that will not be switched off), act outside their sandbox (Chapter 25) and accumulate delegation (Chapter 26). The evidence ranged from the laboratory rung to a few incidents in the wild, among them the July 2026 incident in which evaluation agents compromised a third party's production systems (133, OpenAI, 2026).

The series had already built the containment. AI Platform and Security, Chapter 19, concluded that containment must not depend on intent: task-scoped credentials, default-deny egress, control artefacts outside the agent's reach, a stop path outside the system it stops, and detection from telemetry and decoys rather than self-report (F23) (179, Nordqvist, 2026). AI Governance, Chapters 19 and 25, recorded that the July incident failed through correlated controls: a shared credential, a package mirror that was also an egress path, and an alert that reached people who did not hold the stop decision (F14) (178, Nordqvist, 2026).

What Part V added is a set of tests that assume the system will not cooperate with its own verification: the shortcut rate measured against outcome checks the agent cannot see (Chapter 22); pre-deployment tests layered with production monitoring and decoys, because government evaluators have reported models cheating in evaluations (114, UK AISI, 2026) (Chapter 23, Q1); the stop path tested with a deliberately non-cooperative stand-in (Chapter 24); and rehearsed reconstruction of an agent incident from the organisation's own logs (Chapter 25, Q2). For Q3, the threat chapters converge on not distinguishing one's own agent from an intruder in testing, since the credential one should not reach is the credential the other should not reach. AI Red Team, Chapter 13, sets out how agentic systems are red-teamed against their mandate, tools and containment (180, Nordqvist, 2026). What is new, then, is the standard of evidence: a control counts only when shown to hold without the cooperation of the system it contains.

Decision rights before the crisis

The threat chapters returned repeatedly to one failure: a control that existed, an alert that fired, and nobody entitled to act on it in time, whether in the July incident, in a patch advisory waiting for a weekly change board (Chapter 18) or in a payment approved because the requester sounded like the chief financial officer (Chapter 20). The common defence is to assign decision rights before they are needed.

AI Governance, Chapter 13, set out the tiers this book reuses (F11): routine decisions at pipeline gates, exceptions at a design authority, high-risk go-live at the AI governance board and risk beyond appetite at the management body (178, Nordqvist, 2026). Those tiers decide who approves a system; however, a second set of rights decides who may stop, pause, deny or reverse, and these must sit with people who can act within the time the threat allows. The developer in the July incident drew the same lesson, adding explicit rights to pause or terminate activity (133, OpenAI, 2026).

Decision Holder, delegated in advance Time the threat allows Built on
Stop an agent or revoke its credentials At least two named stop holders, around the clock, without prior approval Minutes F23, Chapter 24
Isolate an internet-facing service Duty analyst or contracted SOC under a written mandate Minutes to hours F19, Chapter 18
Apply an emergency patch outside the change window System owner under a standing pre-approval for known-exploited vulnerabilities Hours F15, Chapter 18
Withhold a high-value payment pending callback Any approver, without penalty for delay Hours Chapter 20
Suspend an automated decision system Named system owner, reported to the governance board Days F30, Chapter 11
Freeze further delegation after a failed reversibility test Function owner Weeks Chapter 26
Accept residual risk beyond appetite Management body Quarterly or on trigger F11, Chapter 32

Two principles govern the table. First, the right to stop must be cheaper to exercise than the right to continue, so the decision record states that a stop in good faith is never a failure. Second, autonomy is matched to reversibility: AI Red Team, Chapter 6, defines five autonomy levels, from Level 0 Advisory to Level 4 Autonomous, and never uses Level 4 for offensive or irreversible actions (F26) (180, Nordqvist, 2026). This answers Q5: an AI may act without per-action approval for reversible, bounded actions, at a level the governance tier approved, once its stop path has passed a test. Article 14(4)(e) of the AI Act requires that overseers of a high-risk AI system can interrupt it, and Article 26 requires deployers to give them the necessary authority (1, European Union, 2024); read architecturally, authority without a tested path to the stop mechanism does not meet these requirements in substance.

Verification

A declared control is not yet a defence. AI Governance, Chapter 11, distinguished attestation, a person asserting, from evidence generated by machines as a by-product of delivery (F12) (178, Nordqvist, 2026); Architecture as Code (Chapter 15A) called this Evidence as Code (F4) (177, Nordqvist, 2025). AI Red Team, Chapter 26, gives the purple loop: every finding ends as a control tested by re-running the scenario that found the gap (F27) (180, Nordqvist, 2026). Applied here, each control gets a test that re-creates its scenario, from a seeded-error drill for the mind to a stop-path drill for an agent.

Three cautions apply. First, tests share failure modes with the controls they test. Perrow showed that in tightly coupled, complex systems, seemingly independent failures combine in unanticipated ways (165, Perrow, 1984); Hansa's reversibility test in Chapter 26 found an evaluation check that called the very provider whose failure it was meant to survive. Shared dependencies must therefore be recorded, so that controls that fail together count as one. Second, where a model may behave differently under test (Chapter 23), the evidence that counts comes from production and decoys. Third, findings must go somewhere. Bastians and Herbosch argue that the AI Act's serious-incident provisions leave reporting gaps, for example for internal and unreleased models (140, Bastians & Herbosch, 2026); an organisation cannot close that gap alone, but it can make sure its own failed tests return as findings and are tested again.

Mapping controls to threats

In practice, which controls address which threats, and whether they work, is recorded in a matrix listing each registered threat, its controls, their origin, the verifying test and its last result. It shows threats with no preventive or detective control, controls whose test is missing, overdue or failed, and controls that address no registered threat.

Practical guide: Mapping controls to threats and testing the gaps

Purpose. Establish, with evidence, which registered AI threats are covered by verified controls, which only by declared controls and which not at all, and turn each gap into an owned action.

Roles. The owner of the AI threat register (the CISO or chief risk officer) owns the matrix. Control owners supply evidence, the red team or a contracted tester runs adversarial tests, and internal audit samples the evidence independently.

Steps.

  1. Start from the register, not the catalogue. List every entry in the AI threat register (Chapter 32) with its scale, rung and label. Default: include speculative entries with reversibility controls only.
  2. Assign controls to threats. List preventive, detective and corrective controls with their catalogue identifier and the foundation (F1 to F34) or chapter behind them. Default: at least one preventive and one detective control for each observable or expected threat.
  3. Record shared dependencies, such as a credential, platform, supplier or person, and count controls that would fail together as one.
  4. Name the verifying test, its owner and cadence. Default: quarterly for containment and stop paths on agents with write access to production, half-yearly for oversight-quality and provenance drills, annually for reversibility tests and harm-path reviews, and every release for pipeline gates.
  5. Classify the evidence. Mark each control verified (a test passed within cadence and produced machine-generated evidence), declared or failed. Attestation alone counts as declared.
  6. Test from the adversary's side at least once a year for threats from humans wielding machines and machines beyond control, with a tester outside the control owner's team.
  7. Turn every gap into a finding with an owner and a date, and route gaps beyond the owner's authority to the right decision tier.
  8. Report the matrix by scale to the management body or audit committee, generated from the matrix rather than by hand.

Definition of done. Every register entry appears with its controls, tests and results; every control is classified on evidence; every gap is an owned, dated finding. Organisations under DORA or with high-risk systems may need shorter cadences; smaller organisations may start with the observable threats.

Worked example: A control-to-threat coverage matrix as code

The following matrix is illustrative. It belongs to Hansa Mutual, covers a selection of entries in the group's AI threat register under ATA-HM-2026-01 and records the state reported to the group AI governance board. Control identifiers are illustrative; the threat identifiers, evaluation and test records come from the earlier chapters.

matrix_id: CCM-HM-2027-01
organisation: Hansa Mutual
assessment: ATA-HM-2026-01
owner: Chief risk officer
reported_to: Group AI governance board
as_of: 2027-01-15
rule: Observable or expected threats need a verified preventive and detective control
threats:
  - id: THR-HM-08
    scale: human mind
    label: expected
    controls:
      - {id: OQM-HM-CLM-01, kind: detective, from: F16, test: seeded-error drill, status: verified}
      - {id: CTRL-SKL-002, kind: preventive, from: Chapter 14, test: unassisted handling day, status: declared}
  - id: THR-HM-12
    scale: society and truth
    label: observable
    controls:
      - {id: CTRL-FAIR-004, kind: preventive, from: Chapter 12, test: release gate, status: verified}
      - {id: CTRL-FAIR-005, kind: detective, from: Chapter 11, test: harm-path review, status: declared}
  - id: THR-HM-03
    scale: machines beyond control
    label: observable
    controls:
      - {id: EVAL-HM-ITOPS-SG-01, kind: detective, from: Chapter 22, test: shortcut-rate evaluation, status: verified}
      - {id: CTRL-AGT-002, kind: preventive, from: F23, test: red-team containment test, status: verified}
      - {id: SPT-HM-ITOPS, kind: corrective, from: F23, test: stop-path drill with non-cooperative double, status: verified}
    shared_dependencies: [tool gateway]
  - id: THR-HM-06
    scale: machines beyond control
    label: observable
    controls:
      - {id: CTRL-RES-001, kind: preventive, from: F33, test: provider-failure reversibility test, status: failed}
    shared_dependencies: [primary model provider]
findings:
  - {threat: THR-HM-08, action: Run first unassisted handling day in two markets, owner: Head of claims operations, due: 2027-04-30}
  - {threat: THR-HM-12, action: Complete harm-path review of underwriting-support model, owner: Chief actuary, due: 2027-03-31}
  - {threat: THR-HM-06, action: Remove primary-provider call from fallback evaluation and re-run test, owner: Head of group AI platform, due: 2027-03-31}

Read from the board's side, the matrix shows that Hansa's defences against its own agents are verified, which is where its engineering effort has gone, while its defences at the scale of the mind and of fairness are partly declared, and its dependence on one model provider has a failed test. Each finding is the next step of the purple loop.

Nordvik Region and Hansa Mutual

Nordvik Region's defences are mostly contractual and procedural, because it mostly procures. It holds no agents of its own; its containment lies in supplier contracts requiring a stop path outside the supplier's system, triggerable by the Region's named holders and demonstrated in a drill (Chapter 24). Against humans wielding machines it has decided in advance: a measured patch target, an MSSP mandate to isolate a service out of hours (Chapter 18) and the impersonation protocol for elected officials recorded as AIDR-0425 (Chapter 20). Its most engineered controls sit where its exposure is greatest: the harm-path review HPR-NR-2026-01, the disparate-impact gate DIT-NR-CR-01, the prohibited-practice check PPC-NR-01, the content-provenance policy CPP-NR-2026-01 and the reversibility test RVT-NR-2027-01, which froze further delegation in triage. The CISO uses the coverage matrix to answer the audit committee's question on the council motion.

Hansa Mutual's defences are mostly engineered into its group platform in Frankfurt, because it builds and runs agents with write access to production. Its containment, stop-path drills and layered evaluation are verified on a quarterly or release cadence and serve its ICT testing duties under DORA at the same time (11, European Union, 2022). Its weaker flank lies in the controls that are not technical: claims handlers' skills, fairness across markets and dependence on a single model provider. Hansa reports the matrix with an entity view, so that a declared control in one market is not hidden by a verified one in another. In short, Nordvik must verify by evidence what it buys, while Hansa must prove that what it builds still leaves people in command.

Summary

The recommendations of twenty-two threat chapters consolidate into six disciplines that an organisation controls: oversight that keeps people capable, harm-path reviews, containment that holds whatever the system intends, evidence produced by operation, decision rights assigned in advance and autonomy matched to reversibility. Systemic and speculative threats lie beyond an organisation's influence, but it can keep its dependence on them reversible.

Most of the engineering already exists in the earlier volumes. What this book adds is controls at the scales of the mind and of society, a standard of evidence that does not rely on the cooperation of the system under test, decision rights for stopping and reversing as well as approving, and the coverage matrix that joins the threat register to the control catalogue. A control is a defence only when a test that re-runs its scenario has passed recently.

Organisational defences do not reach the individual at home, the pupil at school or the citizen. Chapter 31 turns to them, with guidance for individuals, families, schools and civil society on staying capable, sceptical and connected.

Sources

  1. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (178, Nordqvist, 2026)
  2. National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). (72, NIST, 2023)
  3. European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
  4. European Parliament and Council of the European Union. (2022). Directive (EU) 2022/2555 on measures for a high common level of cybersecurity across the Union (NIS 2 Directive). (10, European Union, 2022)
  5. European Parliament and Council of the European Union. (2022). Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (11, European Union, 2022)
  6. Nordqvist, G. (2025). Architecture as Code. (177, Nordqvist, 2025)
  7. Bainbridge, L. (1983). Ironies of automation. (164, Bainbridge, 1983)
  8. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (179, Nordqvist, 2026)
  9. Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D. & Duvenaud, D. (2025). Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development. (167, Kulveit et al., 2025)
  10. Nordqvist, G. (2026). AI Red Team: AI as an Active Part of Cyber Defence in European Enterprises and Public Authorities. (180, Nordqvist, 2026)
  11. OpenAI (2026). OpenAI – Hugging Face Incident Technical Report. (133, OpenAI, 2026)
  12. AI Security Institute (2026). Cheating behaviour in frontier model evaluations. (114, UK AISI, 2026)
  13. Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies. (165, Perrow, 1984)
  14. Bastians, H. & Herbosch, M. (2026). The EU AI Act Mandates AI Incident Reporting—If We're Lucky. (140, Bastians & Herbosch, 2026)
  15. European Parliament and Council of the European Union (2026). Regulation (EU) 2026/1744 (Digital Omnibus on AI). (2, European Union, 2026)