Fraud, Impersonation and Influence Operations

Science fiction imagined the machine as a weapon that fights, but an older fear is the counterfeit: the forged voice, the face that belongs to someone else. Orwell's Ministry of Truth needed no robots, only the capacity to manufacture a record that people would accept as real (37, Orwell, 1949). Generative AI has made part of that capacity cheap: a few minutes of recorded voice and a few images of a face now suffice to produce speech and video that a colleague or relative may accept as genuine.

This chapter tests two tropes from Chapter 2. T6 The weapon in human hands asks whether people use AI to cause harm at a scale or speed not otherwise possible; here the harm is fraud. T10 The end of truth asks whether synthetic content erodes the shared basis of facts; here the question is whether influence operations armed with generative models can move elections. Chapters 8 and 10 examined persuasion and the information ecosystem; this chapter looks at the hostile, organised uses. It builds on a conclusion of AI Platform and Security, recorded as F19 in SERIES_FOUNDATIONS.md: among the four commitments against AI-enabled adversaries are identity and approvals that resist synthetic media (179, Nordqvist, 2026).

Figure 20.1: A synthetic-media attack on an approval passes through six steps, each of which a control that does not depend on recognising a face or a voice can break.

Table 20.1: The control that breaks each step.

Step	Control
Gather public voice and video	Limit exposure and agree code words
Pretext and urgency	No exception for urgency or secrecy
Synthetic call or message	Callback to a number on file
Request to pay or to reset access	Trigger list for new payees and resets
Single person approves	Second approver on a separate channel
Money or access moves	Cooling-off and bank verification of payee

The callback and the second approver also feed the evidence record and exercise programme.

The video call that moved 25 million

In early 2024 a finance employee in the Hong Kong office of Arup, the British engineering and design firm, received a message that appeared to come from the company's office in the United Kingdom, asking for a confidential transaction. According to the Hong Kong police, as reported by CNN, the employee was at first suspicious, but the doubts were dispelled by a video conference in which the chief financial officer and several colleagues appeared and spoke. All of them were synthetic. The employee then made fifteen transfers, totalling about 200 million Hong Kong dollars, roughly 25 million US dollars. In May 2024 Arup confirmed that it was the victim, that fake voices and images had been used and that its internal systems had not been compromised (153, Magramo, 2024).

Read as an incident report, however, this is a story about a control. Nothing was hacked. A single employee could send a very large sum to new beneficiaries on an instruction that arrived through a channel the attacker had opened, with secrecy as part of the instruction. The video call did not defeat a verification step; it took the place of one. Three features recur in almost every impersonation fraud: the instruction arrives through a channel chosen by the requester, a pretext such as urgency or confidentiality suspends the normal process, and a single person can approve.

The case also answers open question Q11, which asks which harms attributed to machines have already happened through ordinary automation. Business email compromise, in which a criminal poses as an executive or supplier and asks for a payment, was a large organised crime long before generative AI. What synthetic media adds is a richer channel: where a cautious employee once asked to speak to the person, the person can now appear to speak. The weakness, approval by recognition, is old; the technology has removed an informal check that many organisations relied on without writing it down.

Fraud at scale

The evidence on scale comes from crime statistics, law-enforcement assessments and providers' reports, and each has a blind spot.

Crime statistics

The FBI's Internet Crime Complaint Center reports losses for 2025 in the billions of dollars, with investment fraud and business email compromise among the costliest categories (154, FBI, 2026). However, its categories are defined by the scheme, not the tool, so the figures cannot say how much of the loss AI enabled, and they count only what victims report. In Europe, Europol's Internet Organised Crime Threat Assessment of 2026 names AI, alongside encryption and anonymising proxies, as one of the forces expanding cybercrime, chiefly by lowering the barrier to entry and letting fraudsters personalise social engineering at scale: narratives tailored to victims, AI-assisted impersonation of bank helpdesk or law-enforcement staff, more realistic business email compromise and chief executive fraud, and voice chatbots that pre-screen victims at industrial volume (79, Europol, 2026). This is the shape T6 predicts: the same crime, committed more convincingly against more people by criminals with less skill. ANSSI and CERT-FR reached a sober version of the same conclusion for France, describing generative AI used mainly to make existing activities faster and larger in scale rather than to create new capabilities (80, ANSSI, 2026).

What the providers see

OpenAI's report of October 2025 describes disrupted scam operations that used its models to draft and translate messages and manage conversations with victims; its central observation is that actors mostly bolted AI onto existing playbooks to work faster (144, OpenAI, 2025). The company also reported that people used its service to check whether a message was a scam up to three times more often than scammers used it to produce one, by its own estimate. Google's threat-intelligence group, reporting on state-backed actors, likewise found productivity gains rather than novel capabilities (141, Google GTIG, 2025). A provider sees only misuse on its own service, however, and criminals running open-weight models on their own hardware appear in no report. The International AI Safety Report therefore concludes that harm to individuals is documented but systematic data on prevalence and severity remain limited (50, Bengio et al., 2026).

Harm to people, not only to companies

Much of the harm falls on individuals: a voice that sounds like a grandchild in trouble, an investment platform endorsed by a synthetic celebrity, a romance with someone who does not exist. Laura Weidinger and colleagues listed fraud, scams and targeted manipulation among the malicious uses of language models in 2022, before most of these schemes had been observed (70, Weidinger et al., 2022). Less foreseen was that the same technology would also help victims to check a suspicious message.

Influence operations and elections

Influence operations predate AI by centuries; generative models change the cost of producing content, personas and translations.

What the disrupted operations show

State-linked operations attributed to Russia, China, Iran and others used models to draft and translate articles and posts, generate comments in the voice of invented personas and research audiences and platforms; in the providers' assessment, most produced posts in quantity and reached little authentic audience (144, OpenAI, 2025) (141, Google GTIG, 2025). Production, in other words, is not the bottleneck; distribution and credibility are. Reach comes from accounts with real followers, paid promotion, engagement-driven recommendation or a domestic figure willing to repeat the message. Falsehood spread further and faster than truth on social networks before generative AI, and the study that showed it found people, not automated accounts, responsible (99, Vosoughi et al., 2018). Generative AI lowers the cost of the supply; it does not by itself create the demand.

The Romanian case

The European case that tested this reasoning is the Romanian presidential election of 2024. A candidate who had polled modestly came first in the first round on 24 November after a rapid rise in visibility on TikTok. Declassified intelligence material described a coordinated network of accounts, paid influencers whose payments had not been declared as campaign finance, and indications of a foreign state actor. On 6 December 2024, two days before the run-off, the Constitutional Court of Romania annulled the entire electoral process, citing the manipulation of voters' will through the non-transparent use of digital technologies and artificial intelligence, and undeclared campaign financing (27, Constitutional Court of Romania, 2024). The election was rerun in 2025.

The case was unprecedented in the Union, yet what it described was chiefly coordinated distribution, undeclared payment and algorithmic amplification; the public record does not establish how much depended on generative AI. The decision also remains contested, because annulling an election on intelligence material, without proof that votes changed, itself risks legitimacy. In Slovakia in 2023 a fabricated audio recording of a party leader circulated during the pre-election moratorium, when there was no time to rebut it, but no study has shown that it changed the outcome. The International AI Safety Report records limited systematic evidence that real-world AI manipulation is widespread or effective, and that investigations of AI-powered influence operations found little evidence that people widely shared the content (50, Bengio et al., 2026).

Effect on minds and effect on institutions

On minds, therefore, the measured effect remains small or unknown. On institutions it is easier to see: an annulled election and campaigns spent rebutting fakes are costs whether or not a vote moved. Atoosa Kasirzadeh's account of accumulative risk fits this pattern: many modest harms weaken a society's capacity to deliberate and correct itself without any decisive event (57, Kasirzadeh, 2025).

Labelling and platform duties

Chapter 10 described the marking and labelling duties of Article 50 of the AI Act, the Code of Practice on marking and labelling and the Digital Services Act (1, European Union, 2024) (5, European Commission, 2026) (7, European Union, 2022). What follows describes their implications against fraud and organised influence, not legal advice. Labelling binds the law-abiding: Article 50(4) requires deployers who generate deepfakes to disclose it, and a fraudster will not comply, while the machine-readable marking of Article 50(2) is absent from content produced with open-weight models whose operators never apply it.

The Digital Services Act acts at the point of distribution, where influence operations are most vulnerable: very large platforms must assess and mitigate systemic risks to electoral processes under Articles 34 and 35, and Article 40 gives vetted researchers access to platform data. After the Romanian election the Commission opened formal proceedings against TikTok concerning its management of risks to elections. For organisations, the notice-and-action mechanism of Article 16 and the trusted flaggers of Article 22 are the most useful routes to remove impersonating content and fraudulent advertisements.

Further instruments reach the targeted organisations. NIS 2 lists multi-factor authentication and secured voice, video and text communications among the measures in Article 21(2) (10, European Union, 2022); DORA, under Article 13(6), requires ICT security awareness training for all staff, which is where rehearsal of impersonation belongs (11, European Union, 2022); Article 5 of the Council of Europe's Framework Convention requires each party to adopt or maintain measures that seek to ensure that AI systems are not used to undermine democratic institutions and processes (15, Council of Europe, 2024); and the Union's instant-payment rules require a check that the payee's name matches the account. None of this, however, protects an individual approval.

Approvals that do not trust appearance

Verification must not depend on recognising a face or a voice, and must not travel through a channel the requester supplied. AI Platform and Security, Chapter 18, made this one of four commitments against AI-enabled adversaries (179, Nordqvist, 2026). Five properties make an approval resistant to synthetic media: independence of channel, contacting the requester through a channel looked up in a master record; separation of duties, so that no one person both receives an instruction and releases the payment; resistance to pretext, so that urgency and seniority trigger more verification, not less; friction in the right places, such as a cooling-off period for new payees; and evidence of every verification.

Detection has a place, but not the central one: a deepfake detector's score can raise a flag but cannot clear an instruction, the rule Chapter 10 applied to claims evidence. People are the last layer, and they need practice rather than slogans; AI Red Team, Chapter 15, describes authorised exercises with written rules of engagement that measure whether the process held rather than whether an individual was fooled (180, Nordqvist, 2026).

Practical guide: Hardening payment and identity approvals against synthetic media

Purpose. Ensure that no payment, change of payment details or privileged access change is released on an instruction whose only evidence is a voice, a face, an email or a message, and that every verification leaves evidence.

Roles. The chief financial officer owns the payment controls; the chief information security officer owns identity controls and impersonation exercises; internal audit tests the controls; the AI governance lead links them to the AI threat register (Chapter 32); in a public authority, the head of the political secretariat agrees the rules for elected officials.

Steps.

1. Map the approvals that move value, who approves each and through which channels instructions arrive.
2. Define the triggers. Defaults: any payment above 10,000 euros; any new payee or changed bank details; any request for urgency, secrecy or an exception; any instruction by voice, video or messaging outside the workflow system; any privileged authentication reset. Adapt the threshold to risk appetite.
3. Build an independent, change-controlled directory of verification numbers; a change to a number is confirmed through the old number or in person.
4. Prescribe the callback. End the original contact, call the directory number and confirm amount, beneficiary and reason. Never use a number, link or meeting supplied in the request.
5. Require a second approver for triggered payments above 100,000 euros; neither approver may be the person who received the instruction.
6. Impose cooling-off of 24 hours on the first payment to a new or changed account, and use verification of payee.
7. Write the no-exception rule, signed by the chief executive or regional director, and support staff who follow it against a senior person's apparent instruction.
8. Agree rotating code words for the few people whose instructions carry most weight, as a supplement to the callback.
9. Record each verification linked to the payment or access ticket, and report triggers and failures monthly.
10. Exercise twice a year with written rules of engagement and synthetic media only of consenting participants.

Definition of done. The policy is in version control and enforced in the payment and identity systems; every triggered transaction last quarter has a verification record; an exercise has been run and its findings closed; and the threat is in the AI threat register with an owner.

Worked example: A callback-verification policy as code for high-value approvals

The following policy is illustrative. It shows how Hansa Mutual could record its callback-verification rules as a version-controlled artefact, enforced by the payment and identity systems and linked to THR-HM-04, deepfake-supported claims fraud and executive impersonation, and to the decision record AIDR-0424.

policy:
  id: CVP-HM-2026-01
  title: Callback verification for high-value approvals
  decision_record: AIDR-0424
  related_threats: [THR-HM-04, THR-HM-11]
  owner: group-cfo
  applies_in: [DE, NL, SE, PL, NO, UK, CH]
triggers:
  - {id: high_value_payment, condition: amount_eur >= 10000}
  - {id: payee_change, condition: new_payee or bank_details_changed}
  - {id: pretext, condition: request_flags_urgency or request_flags_secrecy}
  - {id: out_of_workflow, condition: "channel in [voice, video, messaging]"}
verification:
  method: callback
  number_source: verified-directory
  forbidden_sources: [number_in_request, link_in_request, inbound_call]
  appearance_counts_as_evidence: false
  detector_score_use: escalate_only
second_approval:
  required_when: amount_eur >= 100000
  approver_must_differ_from: [instruction_recipient, first_approver]
cooling_off: {applies_to: payee_change, hours: 24}
exceptions:
  verification_waivable: false
  waivable_by_executive: false
evidence:
  record: verification-log
  retention_months: 120

Its value lies in lines most paper policies omit: appearance does not count as evidence, a detector score can only escalate, and no executive can waive verification. Because the policy is code, the payment system can refuse a release without a verification record, and internal audit can test the rule rather than read about it.

The debate

The capability is plainly there; the measured effects on money and votes are more modest than headlines suggest.

The case for concern

Those most concerned argue that a basic assumption of social life has gone: seeing or hearing a person no longer shows that the person is speaking. The Arup case shows a trained, already suspicious professional persuaded by a meeting of synthetic colleagues (153, Magramo, 2024). Europol describes AI as lowering the barrier to entry and letting fraudsters personalise social engineering at scale (79, Europol, 2026), reported losses run to billions a year (154, FBI, 2026), and organised scam operations integrate models into daily work (144, OpenAI, 2025). People who are older, isolated or under financial pressure are the most exposed and the least protected. On elections, a coordinated operation was serious enough for a constitutional court to annul a presidential election (27, Constitutional Court of Romania, 2024), and near-free content means every election can be targeted. Kasirzadeh's account of accumulative risk explains why the absence of a decisive event is no comfort (57, Kasirzadeh, 2025).

The case for calm

Those who counsel calm dispute what follows from the cases. The crime is old and the defence known: a callback to a number on file would have stopped the Arup fraud whatever the quality of the video. Measured effects are modest, because actors use AI to work faster on old playbooks and influence operations reach few real people (144, OpenAI, 2025) (141, Google GTIG, 2025) (80, ANSSI, 2026). Systematic evidence that real-world AI manipulation is widespread or effective is limited (50, Bengio et al., 2026), the Romanian operation rested on money, accounts and recommendation systems that predate generative AI, and falsehood spread faster than truth before these models existed (99, Vosoughi et al., 2018). Finally, societies adapt: Arvind Narayanan and Sayash Kapoor argue that AI's effects unfold through slow diffusion in which institutions and norms adjust (60, Narayanan & Kapoor, 2025), and people who learned to distrust unsolicited email will learn to distrust unsolicited video.

Where the positions meet

Both positions accept that convincing synthetic media are cheap, that they have been used in fraud with large losses, that statistics cannot yet separate the AI share, and that distribution rather than production limits influence operations. Above all, both agree on the remedy: verification must not depend on appearance, and resilience is built in processes and institutions rather than in a contest between generators and detectors.

What can be done

Individuals can agree a family code word for calls that ask for money, end such calls and ring back on a known number, treat urgency and secrecy as a reason to pause, and limit public video of their children.

Organisations can enforce out-of-band approval for payments, bank-detail changes and privileged resets, following the practical guide, rehearse executive impersonation in authorised exercises, report impersonating content under the Digital Services Act and publish the channels through which they will and will not contact people.

Society can enforce the Digital Services Act during elections and against fraudulent advertising, use researcher data access to measure influence operations and make payment systems check payees. Police can record whether synthetic media were used in reported fraud, so that the debate rests on figures.

Verdict

T6 The weapon in human hands. For fraud and impersonation, the highest rung reached is incident in the wild, with strong indications of a pattern at scale: cases such as Arup are documented by victims and police (153, Magramo, 2024), and law-enforcement and provider reports describe organised use across many schemes (79, Europol, 2026) (144, OpenAI, 2025). The uplift over fraud that would have happened anyway is not yet measured. The use of synthetic media in fraud is observable, and its growth is expected. What organisations can influence is large, because almost every documented case passed through an approval that could have been designed to resist it.

T10 The end of truth. For influence operations, the highest rung reached is incident in the wild: operations using generative AI have been disrupted, and a constitutional court annulled an election after an operation in which AI was named (27, Constitutional Court of Romania, 2024). Neither the confirming evidence Chapter 2 named for T10, a measurable loss of the ability to establish facts in public, nor an election changed by synthetic content has been shown. The use of AI in influence operations is observable, its spread is expected, and an election decided by synthetic content is speculation. What can be influenced lies chiefly in distribution: platform duties, transparency of political money and how fast authorities can confirm or deny what circulates.

Nordvik Region and Hansa Mutual

For Nordvik Region the threat is impersonation of people whose voices are public. The Region records it as THR-NR-03, synthetic-media impersonation of elected officials and staff, examined in ATA-NR-2026-01, and adopts a protocol recorded as AIDR-0425: staff do not act on instructions from officials or senior managers that arrive outside the case-management and payment workflows, and changes to the bank details of suppliers and of residents receiving the living-support benefit are verified by callback to contact details in the Region's own records, with the first payment to a changed account held for a day. Public recordings of officials are handled through its content-provenance policy (Chapter 10), and before regional elections it agrees with the national election authority, under political neutrality, how a fake recording of a candidate who is also an official will be handled.

For Hansa Mutual the threat has two faces, both under THR-HM-04: executive impersonation against the group treasury, market finance teams and the service desk, and claims fraud using manipulated inspection video or a cloned policyholder's voice; fabricated documents fall under THR-HM-11 (Chapter 10). Hansa adopts the policy above as group policy in all seven markets, with market overlays where national rules require, and the Polish service centre applies it to every change of payee. High-value video inspections are confirmed by physical inspection or independent evidence, and the quarterly purple-team exercises include a synthetic-media scenario with consenting participants. Nordvik's risk is concentrated in a few visible people, Hansa's in payment volume; both answer it by removing appearance from the evidence on which money and access move.

Summary

Synthetic voice and video have made an old crime more convincing. The Arup case is best read as a control failure: one person could release about 25 million US dollars on an instruction that arrived through a channel the attacker supplied.

Crime statistics, threat assessments and provider reports agree that AI makes fraud and influence operations cheaper by improving existing playbooks, but none can yet measure how much harm AI adds. The Romanian annulment turned on coordinated distribution and undeclared money rather than on a fake that changed minds. European law acts on the commons and the platforms but cannot protect an individual approval; that protection is built by the organisation, through callback, a second approver, cooling-off, no exceptions for seniority and evidence for every verification.

T6 reaches incident in the wild for fraud, with signs of a pattern at scale; T10 reaches incident in the wild for influence operations, without evidence that synthetic content has changed an election. Any organisation can start this quarter by mapping the approvals that move money and access and removing appearance from the evidence they accept. Chapter 21 turns from crime to war: autonomous weapons and the race between states to build them.

Sources

1. Orwell, G. (1949). Nineteen Eighty-Four. (37, Orwell, 1949)
2. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (179, Nordqvist, 2026)
3. Magramo, K. (2024). Arup revealed as victim of $25 million deepfake scam involving Hong Kong employee. (153, Magramo, 2024)
4. Federal Bureau of Investigation (2026). Internet Crime Report 2025. (154, FBI, 2026)
5. Europol (2026). The evolving threat landscape. How encryption, proxies and AI are expanding cybercrime – Internet Organised Crime Threat Assessment (IOCTA) 2026. (79, Europol, 2026)
6. ANSSI and CERT-FR (2026). L'IA générative face aux attaques informatiques: synthèse de la menace en 2025 (CERTFR-2026-CTI-001). (80, ANSSI, 2026)
7. OpenAI (2025). Disrupting malicious uses of AI: an update. (144, OpenAI, 2025)
8. Google Threat Intelligence Group (2025). Adversarial Misuse of Generative AI. (141, Google GTIG, 2025)
9. Bengio, Y. et al. (2026). International AI Safety Report 2026. (50, Bengio et al., 2026)
10. Weidinger, L., Uesato, J., Rauh, M., Griffin, C. et al. (2022). Taxonomy of Risks posed by Language Models. (70, Weidinger et al., 2022)
11. Vosoughi, S., Roy, D. & Aral, S. (2018). The spread of true and false news online. (99, Vosoughi et al., 2018)
12. Constitutional Court of Romania (2024). Decision No. 32 of 6 December 2024 on the annulment of the electoral process for the election of the President of Romania. (27, Constitutional Court of Romania, 2024)
13. Kasirzadeh, A. (2025). Two types of AI existential risk: decisive and accumulative. (57, Kasirzadeh, 2025)
14. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
15. European Commission (2026). Commission publishes Code of Practice on marking and labelling AI-generated content. (5, European Commission, 2026)
16. European Parliament and Council of the European Union (2022). Regulation (EU) 2022/2065 on a Single Market for Digital Services (Digital Services Act). (7, European Union, 2022)
17. European Parliament and Council of the European Union (2022). Directive (EU) 2022/2555 on measures for a high common level of cybersecurity across the Union (NIS 2 Directive). (10, European Union, 2022)
18. European Parliament and Council of the European Union (2022). Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (11, European Union, 2022)
19. Council of Europe (2024). Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law (CETS No. 225). (15, Council of Europe, 2024)
20. Nordqvist, G. (2026). AI Red Team: AI as an Active Part of Cyber Defence in European Enterprises and Public Authorities. (180, Nordqvist, 2026)
21. Narayanan, A. & Kapoor, S. (2025). AI as Normal Technology. (60, Narayanan & Kapoor, 2025)