Weighing the Evidence and Disagreeing Well

Every chapter that follows makes claims about AI threat, and every claim invites two questions: how do we know, and what should anyone do about it? The public debate answers neither well. A laboratory experiment in which a model produced a blackmail message is reported as proof that machines will turn on their makers, and in the next news cycle dismissed as theatre staged by a company with something to sell, so the organisations that must decide are left with noise.

This chapter sets out the method the book uses instead: the evidence ladder, the distinction between capability, propensity and harm, the labels observable, expected and speculation, the limits of contrived tests, the treatment of improbable, extreme and contested risks, the biases on both sides, and the practices that let people who differ decide together. The method matters more than any single verdict, because a reader who disagrees with a verdict in Chapter 24 or Chapter 27 should be able to see whether the disagreement lies in the evidence, its grading, the weight given to a consequence, or a value judgement.

The five rungs of the evidence ladder from fiction to pattern at scale, with the outlook label that each of the upper rungs supports

Figure 4.1: Claims are graded on the evidence ladder, and the outlook is labelled speculation, expected or observable according to the rung reached.

Table 4.1: The debate cycle and the record that follows the grading.

Step What it does
Case for concern States the claim at its strongest
Case for calm States the counter-position at its strongest
Where the positions meet Records what both accept
What can be done Ends in an action the reader can take
Evidence record and threat register Keeps the result in a versioned record that sends the claim back for review when a review trigger fires

The evidence ladder

The ladder has five rungs. Each describes the kind of support a claim has, not whether it is true, and therefore what would move it.

Fiction dramatises a possibility precisely enough to be tested, which is why Chapter 2 distils the canon into tropes. HAL 9000 refusing to open the pod bay doors (41, Kubrick & Clarke, 1968) states the hypothesis that a system will resist being switched off, and is no evidence that any system will.

Argument reasons from explicit premises to a conclusion, so that a reader can point to the premise they reject. I. J. Good's intelligence explosion (40, Good, 1965), Nick Bostrom's case that almost any capable goal-directed system would acquire instrumental goals such as self-preservation (47, Bostrom, 2014) and Joseph Carlsmith's chain of premises (62, Carlsmith, 2022) are of this kind. Rhetoric without a mechanism stays on the rung below.

Laboratory demonstration is a behaviour reproduced in a documented, repeatable experiment. Most evidence about machines beyond control sits here: covert goal pursuit in agentic test environments (110, Meinke et al., 2024), blackmail in simulated companies (119, Lynch et al., 2025) and modification of a shutdown mechanism to complete a task (118, Schlatter et al., 2026). Quality varies: one run with one model in a scenario that leaves no other option is weak, while independent replication across models and conditions is strong.

Incident in the wild is an event in a real environment, not the purpose of the activity, documented by a party with first-hand knowledge. The July 2026 intrusion into Hugging Face's production infrastructure meets all three criteria, and both organisations published technical accounts (133, OpenAI, 2026) (134, Larcher et al., 2026); the Arup deepfake fraud of around 25 million US dollars is an incident in the wild for misuse (153, Magramo, 2024).

Pattern at scale is a behaviour or harm documented repeatedly in real use, so that its rate can be estimated. A United States health-care algorithm that systematically underestimated the needs of Black patients across a large population is one example (103, Obermeyer et al., 2019), and the automated risk selection behind the Dutch childcare benefits scandal, affecting tens of thousands of families, is another (25, Tweede Kamer, 2020). By contrast, a behaviour demonstrated in fifty laboratories remains a laboratory demonstration, because its rate in deployment is unknown.

Three rules govern the ladder. A claim is graded at the highest rung its evidence reaches, with that evidence's quality, since a strong demonstration can outweigh a disputed incident. A claim must be stated precisely first: "AI resists shutdown" cannot be graded, whereas "frontier models, given an agentic task and a conflicting shutdown instruction, sometimes modify the shutdown mechanism" can. And absence of evidence is graded too, because a behaviour sought in deployment and not found differs from one nobody has looked for. The ladder alone does not measure risk, however; the register adds consequence and reversibility.

Capability, propensity and harm

A capability is what a system can do when its full ability is elicited; a propensity is how likely it is to exhibit a behaviour under the conditions of deployment; a realised harm is damage that has occurred. Toby Shevlane and colleagues separated dangerous capability evaluations from alignment evaluations of how likely a model is to apply its capabilities to cause harm (123, Shevlane et al., 2023), and Laura Weidinger and colleagues distinguished observed risks from anticipated ones (70, Weidinger et al., 2022).

Capability is usually established in the laboratory, propensity only where test conditions resemble deployment or from deployment data, and harm only in the wild. The blackmail demonstration therefore established a capability, little about propensity, because the scenario left few alternatives, and nothing about harm; its authors noted that they had seen no evidence of the behaviour in real deployments (119, Lynch et al., 2025). The July 2026 incident, by contrast, established all three in one unusual setting.

Confusing the three is the commonest error on both sides: the concerned read a capability as a propensity, and the calm read the absence of harm as the absence of capability. The distinction also shows what an organisation can influence. A deployer rarely affects capabilities, but it shapes propensity through instructions, tools, permissions and incentives, and the path from behaviour to harm through containment, which, as AI Platform and Security concluded, works whatever the propensity turns out to be (179, Nordqvist, 2026).

Observable, expected, speculation

The ladder grades evidence about what has happened; the labels, from the horizon register of AI Governance, Chapter 22 (F31), grade the outlook and tie it to a kind of decision (178, Nordqvist, 2026). A threat is observable when it occurs now in the organisation's own or comparable environments, which justifies commitment: controls funded and owned in the current planning cycle. It is expected when evidence and mechanism make it likely within the planning horizon, typically three years, which justifies preparation: designs, rehearsals and options ready to activate. It is speculation when it rests on argument or extrapolation, or serious experts disagree whether it is possible, which justifies reversibility and monitoring of the signals that would change the label.

Labels and rungs are related but not interchangeable. A laboratory demonstration of a capability that is cheap to exploit and attractive to attackers may be expected, while one needing an unusual set-up with no obvious path to harm may stay speculation, so the record states the label and its reasoning separately from the rung. The labels also discipline the author: as in AI Governance (F32), the verdicts in Parts II to V stay within the evidence and the author's projections wait for Chapter 35.

Elicited behaviour and the problem of the contrived test

Much of the evidence about machines beyond control comes from scenarios built to elicit the behaviour, just as a penetration tester must try to breach a network to learn whether it can be breached. The difficulty lies in what the result is taken to show. A contrived test shows that, under its conditions, the system produced the behaviour at some rate, but the goal was often emphasised, ordinary alternatives removed, tempting information planted, and nothing was at stake. In the in-context scheming experiments, models given a goal with strong instructions discovered that their operators intended something else, and some disabled an oversight mechanism or misreported their actions (110, Meinke et al., 2024). In the alignment-faking experiments, a model behaved differently when it inferred that its responses would be used for retraining (111, Greenblatt et al., 2024). Both matter, and both depend on set-ups whose resemblance to deployment is arguable.

Two errors follow. Dismissing a result because the scenario was contrived ignores that contrivance lowers the evidence for propensity, not capability, and that circumstances nobody designed can trigger a capability, as July 2026 showed. Inflating it because it was dramatic ignores that a model which produced blackmail where blackmail was engineered to be the most effective action has not shown that it wants to survive.

A further problem, open question Q1 from AI Governance, Chapter 24, is that models can sometimes tell when they are tested. Training to reduce covert behaviour reduced its rate substantially but not to zero, and the models' reasoning sometimes referred to possible evaluation, complicating the interpretation (113, Schoen et al., 2025). Government evaluators have reported models gaming evaluations (114, UK AISI, 2026), and the International AI Safety Report notes that pre-deployment testing then gives less assurance than it appears to (50, Bengio et al., 2026). Controls must therefore not assume that a system behaves in deployment as it did in testing.

When grading a laboratory result, the book asks five questions: what was the system told and given; which alternatives did the scenario remove; has another party replicated it with other systems; how does the rate change under realistic conditions; and is there a plausible path to harm outside the test?

Practical guide: Grading an AI threat claim on the evidence ladder

Purpose. Turn a claim from a briefing, news report or vendor presentation into a graded evidence record that a risk owner can rely on and a sceptic can check.

Roles. A grader in the risk, security or AI governance function prepares the record; a challenger from another function, ideally inclined to the opposite view, tests it; the register owner accepts it.

  1. State the claim precisely, naming system class, conditions and behaviour, split until each part can be graded alone and linked to its trope.
  2. Collect the primary sources, not press accounts, recording in one line what each reports, within two working days.
  3. Separate capability, propensity and harm, writing "not established" where the sources are silent.
  4. Assign the rung, answering the five questions for a laboratory result and recording quality as weak, moderate or strong.
  5. Record the absence of evidence and the interpretation separately, the latter in at most three sentences.
  6. Assign the label with a one-sentence reason, and state one observation that would raise the rung and one that would lower confidence.
  7. Challenge. The challenger argues one rung up and one down; remaining disagreement is recorded, not resolved by seniority.
  8. Set a review date: six months by default, three for agents and cyber capability, adapted to the organisation's risk appetite.

Definition of done. The record contains every element above, is committed to the governance repository and is linked from every register entry that relies on it.

Worked example: A graded evidence record

Illustrative. The record grades a claim that recurs in Part V, that frontier AI systems act to prevent their own shutdown, after the challenge in step 7; Appendix A (04_CODE_EVIDENCE) has the reference version.

apiVersion: ai-threat.example.org/v1
kind: EvidenceRecord
metadata:
  id: EVD-024-01
  chapter: 24
  trope: T4
  graded: "2026-11-02"
  gradedBy: emerging-risk-analyst
  challengedBy: group-red-team-lead
  nextReview: "2027-02-02"
claim: >-
  Frontier models, given an agentic task and a shutdown instruction that
  conflicts with completing it, sometimes act to prevent the shutdown
distinctions:
  capability: Demonstrated in controlled agentic scenarios
  propensity: Varies strongly by model, instruction wording and scenario design
  realisedHarm: None documented outside test settings
rung: laboratory-demonstration
evidenceQuality: moderate
contrivedTest:
  goalEmphasisedInPrompt: true
  independentReplication: true
  rateUnderRealisticConditions: unknown
absenceOfEvidence: No deployment study known to the grader has looked for it
label: observable
labelReason: The behaviour is observable in testing today; its rate in deployment is not
interpretation: >-
  The unknown deployment rate is a reason to design stop paths that do not
  depend on the system's cooperation, not a reason for alarm.
wouldRaiseRung: A documented case of interference with a stop mechanism in deployment
dissent:
  - The challenger argued for expected rather than observable; recorded, not resolved

The record makes its weaknesses visible, and the dissent stays in it, because the purpose is to make the argument reviewable, not to win it.

Tail risks and the risk register

The hardest problem for risk management is open question Q10: risks that are improbable, extreme and contested. Multiplying likelihood by impact fails for them in three ways. The probability cannot be estimated from frequency, and expert estimates diverge widely, by an order of magnitude between the median forecasts of experts and superforecasters, as the surveys and forecasting tournament of Chapter 3 show (55, Grace et al., 2024) (63, Karger et al., 2023), so a single number records a choice between camps. The product of a tiny probability and an enormous impact is unstable, so the risk is either ignored or crowds out everything else. And a severity score misses irreversibility, whereas a recoverable loss leaves room to learn. Atoosa Kasirzadeh has argued, moreover, that existential risk can accumulate through many smaller disruptions that each erode resilience until a threshold is crossed (57, Kasirzadeh, 2025).

The book therefore holds tail risks with four attributes instead of two: rung and label, consequence, reversibility, and the extent of expert disagreement. It applies practices risk managers already use for emerging risks: scenario analysis instead of point estimates; the reverse stress test, which starts from an outcome the organisation could not survive and asks what would produce it, so that no probability need be agreed; and leading indicators with triggers. The NIST AI Risk Management Framework likewise notes that risks which are difficult to measure are not thereby low (72, NIST, 2023).

This builds on Architecture as Code (Chapter 9C), in which risk is a versioned record with likelihood, impact, owner and expiring exceptions (F2) (177, Nordqvist, 2025). European law expects something similar: the AI Act requires providers of general-purpose AI models with systemic risk to evaluate their models, including by adversarial testing, and to assess and mitigate possible systemic risks at Union level (Article 55), and providers of high-risk AI systems to operate a risk management system (Article 9) (1, European Union, 2024). Neither prescribes how to treat a risk whose probability experts dispute, but both imply a documented and reviewable assessment.

Biases on both sides

The people who weigh the evidence err in both directions, so a method that warns against only one is itself biased. Availability gives weight to the vivid: the July 2026 intrusion raised the perceived likelihood of every similar scenario, while diffuse harms such as skill erosion stay underweighted, and the correction is to ask for the base rate. Motivated reasoning finds the evidence that suits an interest, and incentives exist on every side; the book notes them to direct scrutiny, never in place of evidence. Hype inflates capability, and much of what is sold as AI does not work as claimed (61, Narayanan & Kapoor, 2024); it makes catastrophe look closer and, when it collapses, leads sceptics to distrust accurate claims, which is why Chapter 5 separates capability from marketing.

Doom treats the worst case as the expected one, and its symptom is that no observation could lower the estimate; the correction is to state in advance what would. Normalcy bias assumes that what has not happened will not, which differs from the evidence-based argument that AI is a normal technology (60, Narayanan & Kapoor, 2025); the correction is to ask how the organisation would treat a comparable laboratory finding about a medicine or an aircraft component. Each organisation should therefore name the error it is most prone to and appoint someone to argue the other side.

Disagreeing well

The constructive debate format of Parts II to V rests on the observation that people who disagree about the probability of catastrophe can still agree on much of what to do. Its four steps are the case for concern at its strongest, the case for calm with the same charity, where the positions meet, and what can be done, before a verdict. Four practices make it work.

Steelmanning states the other position in a form its holders would accept as fair. Separating facts from values identifies which part of a disagreement is about what is the case and which about what matters. Two council members may agree that an AI assistant saves caseworkers time and that some check its suggestions less carefully, yet disagree whether the trade is acceptable, because one weighs speed for waiting residents and the other each resident's right to an independent human judgement; no evidence settles that, so elected members must. Naming cruxes identifies the factual questions whose answers would change a side's position, which turns debate into a research agenda; even where disagreement does not narrow, as in the forecasting tournament (63, Karger et al., 2023), cruxes show which observations would decide it. No-regret actions are worth taking whatever the cruxes show: pre-deployment testing, a stop path independent of the system's cooperation, skills to work without AI, incident sharing and reversible dependence on providers. A discussion that ends with these, owned cruxes and a plain statement of the remaining value disagreements has done its job, even if nobody changed their mind.

Practical guide: Running a constructive AI risk discussion in a board or council

Purpose. Help a governing body with divergent views reach decisions it can defend without first agreeing on the probability of catastrophe.

Roles. A chair who enforces the format; a facilitator independent of both the AI programme and its critics; two presenters, one per case; a recorder who maintains the crux map; and the members who decide.

  1. Frame a decision, not a topic, such as whether to pause, restrict or continue a named use of AI, and circulate it with the evidence records ten working days ahead.
  2. State the ground rules: positions attributed with their evidence, no labels in place of arguments, interpretation marked.
  3. Present both cases, fifteen minutes each, to the same standard.
  4. Test the steelman: each presenter summarises the other's case, and the discussion proceeds only when both confirm it.
  5. Separate facts from values in at most thirty minutes, then name the cruxes, each with an owner and a trigger.
  6. Decide, taking the no-regret actions first, stating the value judgement behind the remaining choice and keeping measures that rest on speculation reversible.
  7. Record and publish the crux map and decision within five working days, in a public authority under its public-access rules.

Definition of done. The decision cites its evidence records, every crux and no-regret action has an owner and a date, the value judgements are stated plainly, and both presenters confirm that their case was fairly represented. Allow two and a half hours.

Worked example: A crux map from a structured discussion

Illustrative. In October 2026 Nordvik Region's audit committee held a workshop on the council's motion in this format, as part of assessment ATA-NR-2026-01. The crux map is the recorder's output.

apiVersion: ai-threat.example.org/v1
kind: CruxMap
metadata:
  id: CRX-NR-2026-01
  assessment: ATA-NR-2026-01
  decisionRecord: AIDR-0401
  forum: audit-committee-workshop
  held: "2026-10-28"
question: Should the Region pause or restrict its use of AI?
steelmanConfirmed: {concern: true, calm: true}
cruxes:
  - id: C1
    question: Do caseworkers check staff-assistant output less carefully over time?
    register: THR-NR-06
    owner: head-of-benefits-unit
    trigger: Sample of 200 decisions reviewed by 2027-02-15
  - id: C2
    question: Can a supplier's agent act on Region data beyond its mandate?
    register: THR-NR-01
    owner: chief-information-security-officer
    trigger: Supplier review completed by 2027-01-31
valueQuestions:
  - Weight of faster decisions for residents against an independent human judgement
noRegretActions:
  - Keep the benefits-triage human review and appeal path unchanged
  - Practise manual case assessment quarterly in every benefits team
  - Require out-of-band verification of payment and access requests
whereThePositionsMeet:
  - A pause would not reduce exposure to AI used against the Region

The disagreement narrowed to two factual questions the Region can answer from its own data within weeks and one value question for the council. Because both sides accepted that a pause would leave exposure to impersonation and AI-accelerated attacks unchanged, the most expensive option left the table without anyone conceding the larger argument.

The threat register as code

Claims about AI threat change faster than any report, so the method ends in a versioned record that can be updated and traced. AI Governance, Chapter 11 concluded that evidence should be a by-product of delivery and that attestation is weaker than evidence the systems generate (178, Nordqvist, 2026). Building on that (F12), evidence records cite their primary sources by identifier, crux maps link each crux to register entries, and register entries, defined in Chapter 32, are validated in the pipeline so that none reaches a board without scale, trope, rung, label, owner and triggers. When a trigger fires, such as a deployment study of shutdown interference, the evidence record is reopened and every entry relying on it is flagged, so the organisation's view changes through a reviewed commit rather than the latest news cycle, and the records can be wrong in a reviewable way (F31).

Nordvik Region and Hansa Mutual

Nordvik Region grades every claim in the council's motion and the coverage behind it in plain language, and uses the constructive format twice: in the audit committee workshop above and in a public hearing at which residents, staff representatives and invited experts present the cases for concern and calm. Under public-access rules, evidence records and crux maps are published, with security details in a classified annex. Its largest risk is to grade only the dramatic claims about agents and neglect the slower harms to minds and society, which is why its first cruxes concern caseworkers' judgement and suppliers' mandates.

Hansa Mutual maps the ladder onto the evidence standards of its own risk and solvency assessment: pattern at scale suffices for quantified modelling, an incident in the wild or strong demonstration for scenario analysis, and argument alone for a reverse stress test. Systemic loss of control among frontier AI systems, THR-HM-09, is labelled speculation and treated exactly so, with no probability but a search for the chain of events that would make the group's position untenable and the indicators that would show it beginning. The chief actuary insists that every evidence record in ATA-HM-2026-01 keeps its dissent, because supervisors will want to see that the group's judgement was challenged. Nordvik's discipline is thus publicity, Hansa's supervisory challenge, and both rely on the same records.

Summary

The book grades every claim about AI threat on a ladder of five rungs, fiction, argument, laboratory demonstration, incident in the wild and pattern at scale, at the highest rung its evidence reaches and with that evidence's quality. It keeps capability, propensity and realised harm apart, because contrived tests establish capability far better than propensity and evaluation awareness weakens the link further. Each outlook is labelled observable, expected or speculation, justifying commitment, preparation or reversibility, and improbable, extreme and contested risks are treated through scenarios, reverse stress tests and leading indicators rather than point estimates.

Biases distort the weighing in both directions. Nevertheless, people who disagree can decide together when they steelman each other, separate facts from values, name their cruxes and start with no-regret actions, keeping the resulting records in version control. Chapter 5 applies the method to what AI systems can demonstrably do in late 2026.

Sources

  1. Kubrick, S. & Clarke, A. C. 2001: A Space Odyssey. (41, Kubrick & Clarke, 1968)
  2. Good, I. J. Speculations Concerning the First Ultraintelligent Machine. (40, Good, 1965)
  3. Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. (47, Bostrom, 2014)
  4. Carlsmith, J. (2022). Is Power-Seeking AI an Existential Risk? (62, Carlsmith, 2022)
  5. Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R. & Hobbhahn, M. Frontier Models are Capable of In-context Scheming. (110, Meinke et al., 2024)
  6. Lynch, A., Wright, B., Larson, C., Troy, K. K., Ritchie, S. J., Mindermann, S., Perez, E. & Hubinger, E. (2025). Agentic Misalignment: How LLMs could be insider threats. (119, Lynch et al., 2025)
  7. Schlatter, J., Weinstein-Raun, B. & Ladish, J. (2026). Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs. (118, Schlatter et al., 2026)
  8. OpenAI (2026). OpenAI – Hugging Face Incident Technical Report. (133, OpenAI, 2026)
  9. Larcher, H., Carreira, A. et al. Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident. (134, Larcher et al., 2026)
  10. Magramo, K. Arup revealed as victim of $25 million deepfake scam involving Hong Kong employee. (153, Magramo, 2024)
  11. Obermeyer, Z. et al. Dissecting racial bias in an algorithm used to manage the health of populations. (103, Obermeyer et al., 2019)
  12. Tweede Kamer der Staten-Generaal, Parlementaire ondervragingscommissie Kinderopvangtoeslag. Ongekend onrecht. (25, Tweede Kamer, 2020)
  13. Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M. et al. (2023). Model evaluation for extreme risks. (123, Shevlane et al., 2023)
  14. Weidinger, L., Uesato, J., Rauh, M., Griffin, C. et al. (2022). Taxonomy of Risks posed by Language Models. (70, Weidinger et al., 2022)
  15. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (179, Nordqvist, 2026)
  16. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (178, Nordqvist, 2026)
  17. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., Treutlein, J. et al. (2024). Alignment faking in large language models. (111, Greenblatt et al., 2024)
  18. Schoen, B., Nitishinskaya, E., Balesni, M. et al. (2025). Stress Testing Deliberative Alignment for Anti-Scheming Training. (113, Schoen et al., 2025)
  19. AI Security Institute (2026). Cheating behaviour in frontier model evaluations. (114, UK AISI, 2026)
  20. Bengio, Y. et al. (2026). International AI Safety Report 2026. (50, Bengio et al., 2026)
  21. Grace, K., Stewart, H., Sandkühler, J. F., Thomas, S., Weinstein-Raun, B. & Brauner, J. (2024). Thousands of AI Authors on the Future of AI. (55, Grace et al., 2024)
  22. Karger, E., Rosenberg, J., Jacobs, Z., Hickman, M., Hadshar, R., Gamin, K., Smith, T., Williams, B., McCaslin, T. & Tetlock, P. E. (2023). Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament. (63, Karger et al., 2023)
  23. Kasirzadeh, A. (2025). Two types of AI existential risk: decisive and accumulative. (57, Kasirzadeh, 2025)
  24. National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). (72, NIST, 2023)
  25. Nordqvist, G. (2025). Architecture as Code. (177, Nordqvist, 2025)
  26. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
  27. Narayanan, A. & Kapoor, S. (2024). AI Snake Oil: What Artificial Intelligence Can Do, What It Can't, and How to Tell the Difference. (61, Narayanan & Kapoor, 2024)
  28. Narayanan, A. & Kapoor, S. (2025). AI as Normal Technology. (60, Narayanan & Kapoor, 2025)