Autonomous Weapons and the AI Arms Race

No image of AI risk is better known than Skynet, the defence network in The Terminator that is trusted to run the military's systems and turns them on humanity (42, Cameron, 1984). For the questions states actually negotiate, the image is misleading. The debate about military AI is not about a network that rebels. It is about weapons that select and engage targets without a human decision for each engagement, about who answers in law when such a weapon kills the wrong people, and about a competition that may push everyone to field systems before anyone understands them.

This chapter tests three tropes from Chapter 2. T6 The weapon in human hands is its main subject. T1 The creature that turns on its maker is the trope the public reaches for, and the chapter separates it from T6. T5 The intelligence explosion enters through the race: the argument that whoever first builds a self-improving system gains a decisive military advantage. The chapter stays at the level of evidence, law and policy and describes no weapon design or targeting method. Its practical concern is open question Q12, new in this book: what a single organisation can do about risks that are systemic, international and decided by a few actors.

The spectrum from remotely operated to out-of-the-loop weapons systems, with the five points of meaningful human control and the civilian autonomy levels as an analogue

Figure 21.1: Weapons systems run from remote operation to no human decision after activation; meaningful human control is exercised at five points and has a civilian analogue in the autonomy levels.

Table 21.1: The spectrum and the points of meaningful human control.

Level of autonomy Description
Remotely operated A human selects and engages every target
Human in the loop The system proposes, a human authorises each engagement
Human on the loop The system selects and engages, a human supervises and can abort
Human out of the loop No human decision after activation

Points of meaningful human control: legal review of the weapon before use; limits on target types, area and duration; an informed human decision to activate; the ability to supervise and abort in time; an accountable commander and review afterwards. The civilian analogue is autonomy Levels 0 to 4, with Level 4 never used for irreversible actions.

Skynet and the real debate

Čapek's robots were made as workers and sold as soldiers before they rose against their makers (35, Čapek, 1920): the sale is T6, the rising is T1, the story of Shelley's creature (32, Shelley, 1818) and of Skynet. Chapter 2 noted that The Terminator fuses the two, so that a question about humans using machines is easily answered with an image of machines turning on humans.

The real debate is almost entirely about T6: that a weapon selecting targets by sensor data and software may fail to distinguish a combatant from a civilian, behave unpredictably, leave no person responsible, or escalate a confrontation faster than commanders can follow. Norbert Wiener warned in 1960 that a machine acting faster than people can follow may carry out a purpose they did not truly intend, drawing examples partly from the automation of war (39, Wiener, 1960). Stuart Russell adds that a fully autonomous weapon needs no operator per unit, so a small organisation can deploy it in very large numbers, which gives it some properties of a weapon of mass destruction (48, Russell, 2019). Neither argument requires the machine to rebel.

What has been reported

Autonomy in weapons is not new: air and missile defence systems have engaged incoming projectiles automatically for decades. What has changed is the spread of cheap uncrewed systems, object recognition in sensor data, and electronic warfare, which cuts the operator link and so rewards systems that can complete their task without it. In the war in Ukraine both sides have been widely reported to add autonomous navigation and terminal guidance, and armed forces have been reported to use AI decision support to prioritise targets for human approval. These reports describe AI assisting human decisions at scale, not weapons deciding whom to kill.

The Libya report

The most cited single piece of evidence is the final report of the United Nations Panel of Experts on Libya, published in March 2021 as document S/2021/229, on the fighting around Tripoli in 2020 (156, UN Security Council, 2021). In one paragraph, the Panel states that retreating Haftar-affiliated forces were hunted down and remotely engaged by uncrewed combat aerial vehicles or by lethal autonomous weapons systems such as the STM Kargu-2, a loitering munition made in Türkiye, and that these were programmed to attack targets without requiring a data connection between operator and munition, a true "fire, forget and find" capability (156, UN Security Council, 2021).

In May 2021 the paragraph was widely reported as the first documented case of an autonomous drone killing people without human instruction. The report does not establish that: it does not state that the Kargu-2 killed or injured anyone, nor whether any engagement was autonomous rather than remotely controlled, and the statement about connectivity describes a capability. Read with the method of Chapter 4, it is credible on the presence of autonomy-capable weapons in an armed conflict and silent on whether anyone was killed by a machine's decision.

The evidence therefore shows that militarisation is real and accelerating, but not yet, to an independent investigator's standard, a weapon killing a human target with no human decision in the chain. Militaries rarely publish how their systems are used, however, so that absence is partly an artefact.

Meaningful human control

International humanitarian law already governs every weapon: attacks must distinguish between combatants and civilians, must not cause excessive civilian harm and must be planned with feasible precautions, and Article 36 of Additional Protocol I obliges states party to review new weapons. Because all of this presupposes a human who judges, the debate has organised itself around meaningful human control: humans must make the judgements the law requires, with enough understanding, time and ability to intervene for their control to be more than nominal. As Figure 21.1 shows, that control runs from legal review, through limits on targets, area and duration, the decision to activate and the ability to abort, to an accountable commander afterwards.

The United Nations process

Since 2014 states have discussed lethal autonomous weapons under the Convention on Certain Conventional Weapons, since 2017 in a Group of Governmental Experts that works by consensus. In 2019 it agreed guiding principles, among them that international humanitarian law applies fully and that human responsibility must be retained because accountability cannot be transferred to machines, but consensus has let a few states block a binding instrument. Many states, the International Committee of the Red Cross and the Secretary-General favour a two-tier approach: prohibit autonomous weapons that cannot be used in compliance with international humanitarian law, including those whose effects cannot be sufficiently understood or predicted, and regulate the rest.

Resolution 79/62 of the General Assembly, adopted on 2 December 2024 by a large majority, therefore convened open informal consultations in New York in 2025, open to all states, international organisations and civil society, to complement the work in Geneva (22, UN General Assembly, 2024). It recommends and convenes; it does not bind, prohibit or define. The honest summary in late 2026 is broad agreement on principles, sustained diplomacy and no binding international instrument specific to autonomous weapons.

Where European law stops

Article 2(3) of the AI Act provides that the Regulation does not apply to AI systems placed on the market, put into service or used, with or without modification, exclusively for military, defence or national security purposes, regardless of the type of entity carrying out those activities (1, European Union, 2024). Recital 24 adds that a system developed for military purposes but used for civilian purposes, or for both, falls within the Regulation (1, European Union, 2024), so dual-use systems stay in scope, and general-purpose model obligations do not lapse because one customer is an armed force. The Council of Europe's Framework Convention likewise excludes national defence (15, Council of Europe, 2024), which is why the United Nations process carries the weight it does.

The civilian principle that matches

The series reached a civilian version of the same principle independently. AI Red Team, Chapter 6, set out five autonomy levels, from Level 0 Advisory to Level 4 Autonomous, matched the level to the reversibility and blast radius of the action, and concluded that Level 4 is never used for offensive or irreversible actions (180, Nordqvist, 2026). Open question Q5, raised in AI Platform and Security, Chapter 18, and AI Governance, Chapter 13, asks when an AI may act without per-action approval and who decides (179, Nordqvist, 2026) (178, Nordqvist, 2026). The answer most states have reached in principle for weapons is the one the series reached for civilian systems: autonomy may be bounded and supervised, but the irreversible decision belongs to an accountable person.

The race

The competition to build the systems from which future weapons will be drawn has three positions, each stated here at its strongest.

The case for winning

Leopold Aschenbrenner argued in Situational Awareness in 2024 that systems as capable as skilled researchers are plausible by around 2027, after which automated AI research could produce superintelligence quickly, conferring a decisive military advantage; the democracies therefore had to treat the effort as a national-security project (64, Aschenbrenner, 2024). His core claim is not a case for recklessness: a secure lead gives room to solve safety problems and negotiate restraint from strength, whereas a pause by careful actors would hand the lead to less careful ones. The framing has since entered official policy: the United States' AI Action Plan of July 2025, Winning the Race, includes steps to accelerate adoption of AI across the armed forces and tighten export controls on advanced computing (175, White House, 2025).

The case for deterrence

Dan Hendrycks, Eric Schmidt and Alexandr Wang accept that states will compete and ask how competition can be kept stable. In Superintelligence Strategy they propose Mutual Assured AI Malfunction: because a bid for unilateral dominance through superintelligence would threaten rivals' survival, rivals would be able and willing to sabotage it, and the expectation of sabotage would deter the bid (65, Hendrycks et al., 2025). They add non-proliferation through control of advanced chips and security for model weights, deliberately borrowing the logic of nuclear deterrence.

The case for restraint

The third position holds that a race is itself the danger. Hendrycks, with Mantas Mazeika and Thomas Woodside, had listed an AI race among four sources of catastrophic risk, because competition presses militaries to delegate decisions to automated systems and developers to cut corners on safety (56, Hendrycks et al., 2023). Leading researchers argued in Science that companies racing for capability were spending a small fraction of their resources on safety (51, Bengio et al., 2024), and the scenario AI 2027 makes the dynamic concrete: in its race branch the developers lose control, in its slowdown branch they keep it (58, Kokotajlo et al., 2025). A more sceptical form comes from Arvind Narayanan and Sayash Kapoor, who argue that the effects of AI depend on slow diffusion through institutions (they set military AI aside as a separate case) and that policies such as nonproliferation carry their own risks, so that, on their view, advantage from capability alone arrives slowly (60, Narayanan & Kapoor, 2025). How fast capability will grow rests on T5, which Chapter 27 examines.

The civilian side of a military technology

For most European organisations the military question arrives through suppliers, the few firms that states regard as strategic assets. Since 2024 several frontier developers have been reported to relax prohibitions on military use and to partner with defence agencies, and one developer's own threat report documents a state-sponsored group using a commercial model in an espionage campaign (143, Anthropic, 2025). The relationship can also turn the other way. In March 2026 the US Department of Defense designated Anthropic as a supply-chain risk after talks broke down over the company's request for assurances that its models would not be used for fully autonomous weapons or domestic mass surveillance; in September 2026 a federal appeals court upheld the designation by two votes to one (176, CNBC, 2026). Chapter 15 reported the case for what it shows about state power over developers. Here the point is narrower: a European customer of any developer with defence ties, or defence conflicts, holds a dependency whose terms can change for reasons unrelated to the customer.

Three exposures follow: continuity, since a provider can be restricted or redirected in a crisis; jurisdiction, since a provider with defence ties may face national-security demands on data, both treated in AI Governance, Chapter 6, as questions of model origin (178, Nordqvist, 2026); and policy, where an organisation's rules object to suppliers engaged in weapons development. Moreover, Nordic member states have revived total-defence planning in which civilian bodies carry crisis duties, so the verifiable sovereignty properties of AI Platform and Security, Chapter 13, become what continuity depends on (179, Nordqvist, 2026).

Practical guide: Assessing a supplier's defence exposure

Purpose. Make the defence exposure of each critical AI supplier visible, so that decisions rest on facts rather than headlines. The guide extends the model-origin assessment and the dependency map of Chapter 15, and takes no view on whether defence work is right.

Roles. The supplier-relationship owner prepares the record; the CISO and procurement lead review it; the risk function links it to the AI threat register (Chapter 32); legal counsel checks statements on export controls or sanctions.

Steps.

  1. List the model and hosting providers behind every system rated critical in the dependency map, including models embedded in licensed software.
  2. For each, collect public facts only, with source and date: home jurisdiction, usage policy on military use, announced defence contracts, and any designation, sanction or export restriction.
  3. Write two to four scenarios in which defence exposure could affect the service, such as restriction by a state, redirection of capacity in a crisis, compelled access to data or a conflict with the organisation's own procurement rules.
  4. Rate each scenario on the threat register's scale and name the response: an exit plan, a second source, a contractual clause or explicit acceptance.
  5. Check that every response for a high-impact scenario has been tested within the last twelve months; record any gap as an action with an owner and a date.
  6. Set review triggers: a new designation or sanction, a policy change, a new defence partnership, an armed conflict involving the provider's home state, and at least once a year.

Definition of done. Every critical AI supplier has a dated record with sourced facts, scored scenarios, named responses and review triggers; every high-impact scenario points to a tested exit plan or a risk accepted by the accountable executive; and the records are linked from the AI threat register.

Worked example: A supplier-exposure record

The following record is illustrative. It shows how Hansa Mutual adds defence exposure to its model-origin assessment GEO-2026-017 for the provider behind its claims-correspondence classification, which has a European open-weight second source in Frankfurt under exit plan EXIT-004. The provider is fictional.

# Illustrative supplier-exposure record (Hansa Mutual). Provider is fictional.
record: SXR-HM-2026-03
owner: Head of group AI platform
date: 2026-11-02
links:
  model_origin_assessment: GEO-2026-017
  exit_plan: EXIT-004
  threat_register: [THR-HM-06]
supplier:
  name: Provider A
  home_jurisdiction: US
  second_source: European open-weight model in hansa-dc-frankfurt
facts:
  - fact: Publicly announced contract with home-state defence agency for analysis tools
    source: Provider press release
    as_of: 2026-06-30
  - fact: No current designation, sanction or export restriction affecting the service
    source: Legal check of official lists
    as_of: 2026-10-31
scenarios:
  - id: S1
    description: Service to EU customers restricted after a designation or export measure
    likelihood: low
    impact: high
    response: Switch to second source under EXIT-004
    response_tested: 2026-09-12
residual_risk: Accepted by the chief risk officer for 12 months
next_review: 2027-11-01

The record does not judge the provider; its value lies in the field response_tested, because a scenario whose response has never been exercised is a hope, not a control.

The debate

This book can weigh the arguments on these subjects, not settle them.

The case for concern

The case for concern begins with the law. Those who hold it argue that a weapon selecting targets by pattern recognition cannot reliably judge distinction and proportionality where combatants and civilians look alike, and that when it errs responsibility becomes diffuse: the commander did not choose the target, the programmer did not foresee the situation, and the machine cannot be punished. The International Committee of the Red Cross and many states see this accountability gap as a reason to prohibit unpredictable systems.

Weapons that put no soldier at risk may also lower the political cost of force, and weapons that need no operator per unit become scalable instruments of killing (48, Russell, 2019); autonomy-capable weapons have already reached a civil war under an arms embargo (156, UN Security Council, 2021). The race, moreover, presses militaries to delegate and developers to deploy early (56, Hendrycks et al., 2023) (51, Bengio et al., 2024). Tightly coupled automated systems fail in ways no one designed, as Perrow showed (165, Perrow, 1984) and the flash crash of 2010 showed for trading algorithms (166, Kirilenko et al., 2017); a flash escalation between armed systems would not reverse itself. Binding rules are absent, European instruments stop at the military boundary (1, European Union, 2024) (15, Council of Europe, 2024), and official race framing makes restraint harder (175, White House, 2025).

The case for calm

The case for calm begins with what autonomy can improve. Some roboticists, military lawyers and ethicists argue that machines do not act from fear, fatigue or revenge, that sensors can in some settings discriminate better than a frightened soldier, and that automatic missile defence has protected people for decades; the question is therefore how autonomy is bounded. Law already applies, the question is on almost every state's agenda (22, UN General Assembly, 2024), and the best-documented case, read carefully, does not show a machine deciding to kill (156, UN Security Council, 2021).

On the race, deterrence has coincided with eight decades without general war between nuclear powers, and mutual vulnerability could stabilise AI competition similarly (65, Hendrycks et al., 2025); a secure democratic lead is a serious strategic position (64, Aschenbrenner, 2024), and slow adoption gives states time to agree rules (60, Narayanan & Kapoor, 2025). Rivals have agreed on risk before: the United States and China both endorsed the Bletchley Declaration (19, UK Government, 2023).

Where the positions meet

Both positions accept that international humanitarian law applies in full, that a human must remain accountable for lethal force, and that unpredictable systems should not be used; the two-tier approach is common ground for many. Both accept that confidence-building measures and incident channels between rivals reduce catastrophic risk, which deterrence also needs (65, Hendrycks et al., 2025), and both want dangerous capabilities kept from non-state actors. For civilian organisations both lead to the same point: know your suppliers' defence exposure and be able to continue if it changes.

What can be done

Individuals can read the short primary sources rather than headlines (156, UN Security Council, 2021), recognise dual-use work and ask their governments what position they take in the United Nations consultations.

Organisations can follow the practical guide above, decide in advance which public functions must run without foreign-hosted models, and keep Level 4 Autonomous away from irreversible actions, as AI Red Team, Chapter 6, requires (180, Nordqvist, 2026), which practises in civilian life the principle states are negotiating for weapons.

Society can press for a binding two-tier instrument on human control over lethal force, use the General Assembly consultations where consensus blocks progress (22, UN General Assembly, 2024), and support incident channels between the major powers. The UN Advisory Body's call for inclusive global governance of AI applies with particular force, since the states most exposed to autonomous weapons are often not those that build them (21, UN Advisory Body, 2024). Chapter 28 describes what has been decided so far.

Verdict

T6 The weapon in human hands. The militarisation of AI, from targeting decision support to uncrewed systems with autonomous navigation and guidance, has reached incident in the wild and is observable. A UN panel recorded weapons able to engage without an operator link in use (156, UN Security Council, 2021), but a machine selecting and killing a human target with no human decision has not been documented to independent standards; wider use of such autonomy is expected. That autonomous weapons will become scalable instruments of mass killing rests on argument and is speculation as a forecast.

T1 The creature that turns on its maker. No evidence suggests that a military AI system has acted against its operators from anything resembling intention, and the Skynet form of the trope remains fiction. The risk that interacting automated systems escalate a confrontation no one chose rests on argument, supported by analogous failures in domains such as finance; as a forecast of war it is speculation, but the failure mode is real.

T5 The intelligence explosion. That the race is shaping state policy is observable in official strategy (175, White House, 2025). That the first state to build a self-improving system will gain a decisive military advantage rests on argument and remains speculation; Chapter 27 weighs the underlying claim.

What can be influenced. No organisation can decide whether autonomous weapons are banned, but every organisation can see its suppliers' defence exposure, keep tested alternatives and keep irreversible decisions with accountable people.

Nordvik Region and Hansa Mutual

Nordvik Region has duties in national civil preparedness, because its health and social-care coordination and public transport must continue in a crisis, and its exposure to military AI runs through suppliers. The dependency map DEP-NR-2026-01 and exit plans EXIT-NR-01 to EXIT-NR-03 of Chapter 15 already address THR-NR-05; preparedness planning adds one rule: every AI service is either not needed in a crisis or able to run on capacity the Region controls. The benefits-triage system runs in its own data centre, hosted workloads use open-weight models on reserved Nordic capacity, and the staff assistant degrades to working without AI, which the Region rehearses.

Hansa Mutual's emerging-risk team added defence designations of providers to the criteria of GEO-2026-017, recorded per provider as in the worked example and linked to THR-HM-06, correlated failure of a shared model provider; exit plan EXIT-004 makes a designation survivable. As an insurer it also considers under THR-HM-07 whether its portfolios could accumulate AI-related losses from a conflict. Nordvik plans for continuity of public services; Hansa for the commercial consequences of a supplier caught in a geopolitical dispute.

Summary

The public image of military AI is Skynet; the real debate is about humans using machines and about who is accountable when weapons that engage without a human decision err. The UN Panel of Experts on Libya recorded autonomy-capable loitering munitions in use in 2020, but did not establish that any engagement was autonomous or lethal. States agree that international law applies and human responsibility must be retained, but no binding instrument exists, and the AI Act, under Article 2(3), excludes systems used exclusively for military purposes while keeping dual-use systems in scope.

The race is real as policy. Its advocates argue that a secure lead gives room for safety and that mutual vulnerability can deter; its critics that racing presses everyone to delegate and deploy too early. For a European organisation the consequence lies in its suppliers, and a dated record of each critical supplier's defence exposure, with tested exits, answers that part of Q12. Militarisation is observable and wider autonomy expected; the Skynet form of T1 remains fiction; a decisive advantage from an intelligence explosion remains speculation. Chapter 22 opens Part V, on machines beyond control, with systems that do exactly what they were asked, and not what was meant.

Sources

  1. Cameron, J. (1984). The Terminator. (42, Cameron, 1984)
  2. Čapek, K. (1920). R.U.R. (Rossumovi Univerzální Roboti). (35, Čapek, 1920)
  3. Shelley, M. W. (1818). Frankenstein; or, The Modern Prometheus. (32, Shelley, 1818)
  4. Wiener, N. (1960). Some Moral and Technical Consequences of Automation. (39, Wiener, 1960)
  5. Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. (48, Russell, 2019)
  6. United Nations Security Council (2021). Final report of the Panel of Experts on Libya established pursuant to Security Council resolution 1973 (2011) (S/2021/229). (156, UN Security Council, 2021)
  7. United Nations General Assembly (2024). Resolution 79/62: Lethal autonomous weapons systems. (22, UN General Assembly, 2024)
  8. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
  9. Council of Europe (2024). Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law (CETS No. 225). (15, Council of Europe, 2024)
  10. Nordqvist, G. (2026). AI Red Team: AI as an Active Part of Cyber Defence in European Enterprises and Public Authorities. (180, Nordqvist, 2026)
  11. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (179, Nordqvist, 2026)
  12. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (178, Nordqvist, 2026)
  13. Aschenbrenner, L. (2024). Situational Awareness: The Decade Ahead. (64, Aschenbrenner, 2024)
  14. The White House (2025). Winning the Race: America's AI Action Plan. (175, White House, 2025)
  15. Hendrycks, D., Schmidt, E. & Wang, A. (2025). Superintelligence Strategy: Expert Version. (65, Hendrycks et al., 2025)
  16. Hendrycks, D., Mazeika, M. & Woodside, T. (2023). An Overview of Catastrophic AI Risks. (56, Hendrycks et al., 2023)
  17. Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P. et al. (2024). Managing extreme AI risks amid rapid progress. (51, Bengio et al., 2024)
  18. Kokotajlo, D., Alexander, S., Larsen, T., Lifland, E. & Dean, R. (2025). AI 2027. (58, Kokotajlo et al., 2025)
  19. Narayanan, A. & Kapoor, S. (2025). AI as Normal Technology. (60, Narayanan & Kapoor, 2025)
  20. Anthropic (2025). Disrupting the first reported AI-orchestrated cyber espionage campaign. (143, Anthropic, 2025)
  21. CNBC (2026). U.S. appeals court upholds Pentagon designation of Anthropic as supply chain risk. (176, CNBC, 2026)
  22. Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies. (165, Perrow, 1984)
  23. Kirilenko, A., Kyle, A. S., Samadi, M. & Tuzun, T. (2017). The Flash Crash: High-Frequency Trading in an Electronic Market. (166, Kirilenko et al., 2017)
  24. UK Government (2023). The Bletchley Declaration by Countries Attending the AI Safety Summit, 1–2 November 2023. (19, UK Government, 2023)
  25. United Nations High-level Advisory Body on Artificial Intelligence (2024). Governing AI for Humanity: Final Report. (21, UN Advisory Body, 2024)