The Automated State: When Public Authorities Got It Wrong

This chapter is about a dystopia that has already happened. In the Netherlands and in Australia, public authorities used automated risk selection and calculation to decide whom to suspect and pursue for money. Large numbers of people on low incomes were treated as cheats on the strength of a flag, a score or an average, and for years the institutions meant to serve them would not listen. None of these systems was advanced AI.

The chapter tests T7 The watching state and T8 The machine that stops, and the humans who forgot, here in its institutional form of officials who stopped checking because the system had already decided. It answers the series' open question Q11 for one class of harm and turns the lessons into a review any authority can run. It builds on F30 from AI Governance, Chapter 17: in public authorities legality comes before capability, the line between decision support and automated decision is drawn in the data model and the audit log, and every case has a reasons log (178, Nordqvist, 2026). That chapter described the controls; this one shows the failures they answer.

The harm path of an automated decision system from data to redress, with feedback to the model

Figure 11.1: The harm path runs from data through a model or rule, a flag, an official action and a burden on the citizen to redress or its absence; complaints, appeals and inquiries feed back to the model only if someone listens.

Table 11.1: The control that blocks each step.

Step Control
Data from registers and applications Legal basis and excluded features
Risk model or rule Published model, FRIA and parity tests
Flag, score or calculated debt Bands and reasons, never a bare score
Official action on the case Accountable official and reasons log
Burden placed on the citizen Burden stays with the authority
Redress or no redress Independent appeal and stop decision

SyRI: a court stops a risk model

System Risk Indication (SyRI) was a legal instrument in Dutch social-security legislation that allowed public bodies to pool data on residents' work, taxes, benefits, housing and debts and run it against a risk model to detect fraud; flagged records became risk reports that could lead to an investigation. The court observed that it had been applied in neighbourhoods described as problem districts.

A coalition of civil-society organisations, including the Dutch section of the International Commission of Jurists (NJCM), joined by a trade-union federation and individual citizens, challenged the legislation, supported by the United Nations Special Rapporteur on extreme poverty and human rights. On 5 February 2020 the District Court of The Hague held that the legislation did not meet the requirements of Article 8(2) of the European Convention on Human Rights, the right to respect for private life, and declared it unlawful and without binding effect (24, District Court of The Hague, 2020). The court accepted that combating fraud is a legitimate aim, but held that the state bears a special responsibility to balance the benefits of new technology against the interference with private life. Because the risk model and its indicators were not public, and the state declined to disclose them, neither the people affected nor the court could verify how the system worked, and the legislation lacked safeguards against unintended discriminatory and stigmatising effects.

SyRI is the cleanest of the cases: a court stopped a structurally unlawful system before it produced harm at scale, and the government did not appeal. AI Governance, Chapter 18 drew the lesson that evidence an outsider can read is part of what makes such a system lawful (178, Nordqvist, 2026).

Unprecedented injustice: the Dutch childcare benefits scandal

The childcare benefits scandal, the toeslagenaffaire, was larger, longer and far more damaging. Dutch parents who use registered childcare receive an allowance, paid in advance and reconciled afterwards by the tax administration's benefits department, which took a hard line on fraud from the early 2010s (25, Tweede Kamer, 2020). Tens of thousands of parents were wrongly treated as fraudsters and required to repay everything, often for an administrative shortcoming such as a missing signature, and many were left with debts of tens of thousands of euros and described losing homes, jobs and relationships (31, Amnesty International, 2021).

A parliamentary committee of inquiry published its report, Ongekend onrecht, "unprecedented injustice", on 17 December 2020 (25, Tweede Kamer, 2020). It found that fundamental principles of the rule of law had been violated, with causes in all three branches of the state: a scheme with little room for proportionality; an executive that applied an all-or-nothing interpretation, withheld information from parents and parliament and did not correct course; and a highest administrative court that upheld the strict interpretation until 2019. The government resigned in January 2021 (31, Amnesty International, 2021).

Automation was part of the story, but not its centre in the inquiry's account. A risk-classification model selected applications for scrutiny, with nationality among the factors that could raise the risk. Amnesty International's Xenophobic Machines reported that the model was self-learning, that officials had little insight into why it had selected cases, and that the combination amounted to discriminatory profiling, and called for a ban on nationality and ethnicity in risk scoring for law-enforcement purposes, including fraud detection (31, Amnesty International, 2021). The Dutch data protection authority separately found the processing of nationality unlawful and discriminatory, and later fined the tax administration. The model, whose mechanism Chapter 12 examines, made the injustice selective and fast; law, a culture of presumed guilt and a political climate that rewarded severity made it total and lasting.

Robodebt: income averaging and the Royal Commission

The Australian Robodebt scheme shows that such harm needs no model at all. From 2016 the Department of Human Services, through Centrelink, compared the income welfare recipients had reported fortnight by fortnight with annual tax-office data and, where they did not match, spread the annual figure evenly across the fortnights, a technique known as income averaging, to infer an overpayment. Recipients had to disprove the debt, often years later and without payslips, and the scheme removed most of the officers who had previously investigated discrepancies.

For people with irregular earnings, such as casual and seasonal workers, averaging produced debts that did not exist. The government conceded in 2019 that debts based on averaging alone had not been validly raised, and later settled a class action that returned money to hundreds of thousands of people. The Royal Commission reported on 7 July 2023 and described the scheme as "a crude and cruel mechanism, neither fair nor legal" (26, Royal Commission, 2023). It found that doubts about its lawfulness had been available inside government from the start, that it reversed the burden of proof, and that warnings from staff, advocates, the ombudsman's office and tribunals went unheeded; families linked the deaths of relatives to the debts. It recommended a consistent legal framework for automated decision-making in government, with a path for review, public information about the business rules and algorithms used, and a body to audit such automation. The calculation was arithmetic; its power came from letting it stand in for evidence.

The common mechanisms

The three cases differ in country, law and technique, but four mechanisms recur, and they are the mechanisms of T7.

Reversed burden of proof. In each case the system's output became a presumption the citizen had to rebut, although administrative law in most European states places the burden of establishing the facts on the authority.

Opacity. The people affected, and often the officials, could not see why someone had been selected or how a figure had been calculated; in SyRI the court itself could not inspect the model (24, District Court of The Hague, 2020), and in the childcare case caseworkers had little insight into its selections (31, Amnesty International, 2021).

Scale. A caseworker who misreads a rule harms the cases on their desk, whereas a system harms every case so consistently that the error looks like policy. Virginia Eubanks found the same pattern in automated eligibility, child-welfare screening and homelessness services in the United States, where systems profiled poor people more intensively than anyone else and made it harder for them to contest the result (169, Eubanks, 2018).

Institutional denial. The signals were there in every case, but the institutions treated them as noise, and both the inquiry and the Royal Commission found that information was withheld from those who could have stopped the harm (25, Tweede Kamer, 2020) (26, Royal Commission, 2023). This is where T8 enters: the humans who forgot were officials who stopped asking whether the system was right, in organisations that had removed the people whose job was to ask. Lisanne Bainbridge observed that automation leaves humans to intervene in the rare moments when a process fails, which are the hardest (164, Bainbridge, 1983); here, nobody was in place to notice.

In the author's reading of the judgment and the reports, summarised in the table, in every case the technology made the failure fast and hard to see, law and culture made it severe and durable, and the correction came from outside.

Case What the technology did What law, politics and culture contributed What stopped it
SyRI Pooled data and ran an undisclosed risk model Legislation without transparency; fraud control focused on poorer neighbourhoods A civil-society challenge and a court
Childcare benefits Selected applications, with nationality as a risk factor An all-or-nothing scheme upheld for years; pressure for severity; information withheld Parents, lawyers, journalists, members of parliament and an inquiry
Robodebt Inferred debts from averaged annual income A method never lawful as sole basis; savings targets; warnings unheeded Advocates, tribunals, litigation and a Royal Commission

What the AI Act now requires

The AI Act, Regulation (EU) 2024/1689, classifies as high-risk, under Annex III point 5(a), AI systems intended to be used by or on behalf of public authorities to evaluate eligibility for essential public assistance benefits and services, and to grant, reduce, revoke or reclaim them (1, European Union, 2024). Under Article 6(3) an Annex III system that profiles natural persons is always high-risk, so a SyRI-type model cannot escape by being called preparatory. The Digital Omnibus on AI postponed the high-risk obligations for Annex III systems to December 2027 (2, European Union, 2026).

The obligations answer the failures directly: risk management (Article 9), data governance that examines data for biases likely to lead to prohibited discrimination (Article 10), transparency to deployers (Article 13) and oversight design that counters automatic reliance on the output (Article 14). Deployers must assign competent overseers and, under Article 26(11), inform people that a high-risk system is used; public bodies must carry out a fundamental rights impact assessment (FRIA) before first use (Article 27) and register their use (Article 49). Article 86 gives a person affected the right to clear and meaningful explanations of the system's role, Article 85 lets anyone complain to the market surveillance authority, and Article 5(1)(c) prohibits social scoring that leads to detrimental treatment in unrelated contexts or out of proportion to the behaviour scored.

Three caveats apply. First, Article 3(1) defines an AI system by its capacity to infer how to generate outputs, so a fixed calculation such as Robodebt's averaging would probably not qualify; it would be caught by Article 22 of the General Data Protection Regulation on decisions based solely on automated processing, and by the right to meaningful information about the logic involved in Articles 13 to 15 (6, European Union, 2016), and national administrative law. Second, SyRI was struck down under the European Convention on Human Rights, so older protections work when someone invokes them. Third, the Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law commits states to keep AI activities consistent with human rights and the rule of law and to provide effective remedies (15, Council of Europe, 2024). None of these duties prevents a harm path by itself; they require the authority to show, in records an outsider can read, that it has blocked it.

The quiet dystopia as a design failure

The automated state rarely announces itself: there is a queue that grows, a letter that does not explain and an official who says the system has decided. It is therefore a design problem. AI Governance described the controls (178, Nordqvist, 2026): a reasons log for every case (Chapter 17); human oversight designed rather than declared (F16), with a blind sample handled without the score and alerts when overrides are too rare as well as too frequent (Chapters 10 and 21); an appeal path handled by people who do not see the model's output; a tested stop decision; and reclassification triggered by change (F10, Chapter 8).

Each control answers a mechanism. The reasons log and the Article 86 explanation answer opacity, letting applicants correct register data before a decision answers the reversed burden of proof, and parity monitoring and the stop decision answer scale. Institutional denial is hardest to design against; the nearest control is to treat complaints, appeal outcomes and ombudsman enquiries as monitoring data with thresholds, routed to someone with authority to stop the system.

Practical guide: Running a harm-path review of an automated decision system

Purpose. Find every path by which an automated decision system could harm the people it serves, name the control that blocks each path and its evidence, and record what remains open.

Roles. The system owner leads, with legal counsel, the data protection officer, the AI governance lead and two front-line officials who describe how the system is actually used; where possible, a legal-aid organisation or the ombudsman's liaison reviews the draft.

Steps.

  1. Draw the path of Figure 11.1 for the system as used, including every downstream use of the output, on one page agreed with front-line officials.
  2. Ask four questions at each step. Can the output shift the burden of proof? Can the person, the official and an appeal body see why? How many people would an error reach before someone with authority notices? Who would hear the signals, and could they stop the system?
  3. Write each harm path as a sentence naming the mechanism, the affected group and the harm.
  4. Name the control and its evidence: a test result, log query, parity report or decision record. A control without evidence is a gap.
  5. Size an error at scale. If a systematic error on the highest-volume path could reach more than one per cent of monthly cases before detection, add monitoring.
  6. Test the stop decision quarterly, with a fallback that can run for at least a month.
  7. Set the signals of denial. Default: an appeal reversal rate above twice the pre-system rate triggers a review.
  8. Record and link the review beside the FRIA and the threat-register entry, and publish a summary with the algorithm register entry.
  9. Repeat on change and at least once a year.

Definition of done. Every path has a control with evidence or is an open gap with an owner and a date, the stop decision has been tested, and the review is linked to the FRIA and the threat register. A system that can reduce or reclaim benefits needs stricter thresholds than one that only orders a queue.

Worked example: A harm-path review of Nordvik's benefits-triage system

The following review is illustrative. It covers Nordvik Region's benefits-triage system, procured from TriageSoft and deployed under Annex III point 5(a), whose claims ranker orders about 40,000 claims a month and the manual verification queue for the living-support benefit. It implements the controls for THR-NR-04 in ATA-NR-2026-01.

harm_path_review:
  id: HPR-NR-2026-01
  status: illustrative
  system: benefits-triage system, claims ranker 4.0.0
  owner: Head of Benefits Unit
  threat_register_entry: THR-NR-04
  linked_records: [AIDR-0007, AIDR-0011, AIDR-0018, FRIA-2026-002, NR-ALG-2026-003]
  paths:
    - id: HP-1
      mechanism: scale
      harm: proxy features raise scores for residents of some areas or one age band
      control: postcode features removed; quarterly parity tests; retirement criterion
      status: closed
    - id: HP-2
      mechanism: reversed burden of proof
      harm: caseworkers treat a high rank in the verification queue as a presumption of error
      control: band and three reasons instead of a score; never auto-accept
      status: controlled
    - id: HP-3
      mechanism: reversed burden of proof
      harm: an applicant must disprove wrong register data
      control: applicant sees and can correct register data before the decision
      status: controlled
    - id: HP-4
      mechanism: opacity
      harm: an appellant cannot learn the system's role in the decision
      control: reasons log per case; Article 86 explanation; appeal unit without the score
      status: controlled
    - id: HP-5
      mechanism: institutional denial
      harm: complaints and appeal reversals are not connected to the system
      control: monthly complaint and reversal report to the system owner and CISO
      status: open
      due: 2027-03-31
  stop_decision:
    who: [senior caseworker, head of social services, AI platform on-call]
    fallback: queue reverts to date order

The review closes one path, controls three and leaves one open, and says so. HP-1 was closed before it caused harm: when a parity gap between age bands persisted beyond the retirement criterion set in AIDR-0007, the Region replaced the model with version 4.0.0 without postcode features, as AI Platform and Security, Chapter 5 records (179, Nordqvist, 2026). HP-5 is the honest gap, the missing connection to outside signals that let the Dutch and Australian failures run for years.

The debate

The case for concern

The case for concern begins with the record. Two democracies with independent courts, free media and strong administrative traditions treated large numbers of their poorest residents as suspects, and kept doing so for years after the signals were visible (25, Tweede Kamer, 2020) (26, Royal Commission, 2023).

Automation also changes the economics of suspicion. Investigating a person used to cost an official's time, which limited how many people the state could suspect; a model removes that limit and lets the institution hide behind the machine. Eubanks described a digital poorhouse (169, Eubanks, 2018), and O'Neil showed how such systems create feedback loops in which those scored most harshly have least means to correct the score (168, O'Neil, 2016).

The concern also looks forward. Arvind Narayanan and Sayash Kapoor argue that predictive systems used to decide about individuals often perform little better than simple baselines while carrying an aura of precision that discourages challenge (61, Narayanan & Kapoor, 2024), and a general-purpose model that summarises the file, recommends the outcome and drafts the letter could make every step more fluent. As Daron Acemoglu and Simon Johnson argue, who benefits from a technology depends on who controls its direction, and in benefits administration the people affected rarely do (67, Acemoglu & Johnson, 2023).

The case for calm

The case for calm reads the same record differently: the decisive failures were of law, politics and culture. Robodebt's arithmetic would have been harmless had a mismatch prompted a request for evidence, and the childcare scandal was driven by an all-or-nothing legal interpretation and a political demand for severity that would have ruined families with paper files; the inquiry located the core of the injustice there (25, Tweede Kamer, 2020). Blaming the machine risks banning tools while leaving the culture intact.

The correction also worked, slowly but decisively. A Dutch court struck down SyRI on the basis of a convention signed in 1950 (24, District Court of The Hague, 2020), and a Royal Commission recommended a legal framework for automated decisions (26, Royal Commission, 2023); Europe has since adopted high-risk duties, the FRIA and the right to explanation (1, European Union, 2024), and the Council of Europe a convention on AI and the rule of law (15, Council of Europe, 2024). Moreover, human caseworkers are inconsistent and their biases hard to measure, whereas a documented system can be tested for parity, audited and corrected. Narayanan and Kapoor's view of AI as a normal technology, whose effects depend on slow institutional adaptation, points the same way (60, Narayanan & Kapoor, 2025): the question is under which conditions to automate.

Where the positions meet

Both positions accept the facts found by the court, the inquiry and the Royal Commission: the technology was simple, law, politics and culture caused much of the harm, automation made it faster and harder to see, and correction came from outside. They agree on the decisive controls, designed in before go-live: a legal basis for what may be automated, usable explanation, a responsible human with authority to depart from the system, and an appeal route independent of it. They differ on how much weight to give more capable AI, which can be tested by watching what authorities do before December 2027.

What can be done

Individuals can ask for an explanation of a high-risk AI system's role under Article 86 of the AI Act (1, European Union, 2024) and for meaningful information about the logic of automated processing under the GDPR (6, European Union, 2016), appeal, and complain to the ombudsman or data protection authority. Officials can record disagreements with a system's output, since in each case staff saw the problem early.

Organisations can run and repeat a harm-path review of each automated decision system, keep a reasons log for every case, design oversight with a blind sample and override alerts, connect complaints and appeal outcomes to the system owner, give a named person a tested procedure to stop the system, and record the risk in the AI threat register (Chapter 32).

Society can fund and protect the institutions that uncovered these failures, from courts, ombudsmen and audit offices to legal-aid organisations, civil-society litigants and investigative journalists, because they work even when the institution itself has stopped listening.

Verdict

T7 The watching state. The trope's core claim, that automated observation and judgement can let a state profile and harm people while escaping accountability, reaches pattern at scale: courts and inquiries in two countries documented harm to tens of thousands of people over several years, and studies in the United States document the same pattern (169, Eubanks, 2018). It is observable, although the machines were not intelligent and courts, parliaments and commissions eventually held the power to account. That more capable AI will make such failures more frequent where authorities automate without the controls described here is expected; the fully unaccountable machine state of Nineteen Eighty-Four remains fiction in Europe (37, Orwell, 1949).

T8 The machine that stops, and the humans who forgot. The institutional form of the trope, that organisations stop checking an automated system's output and so cannot notice when it is wrong, reached incident in the wild in each of the three cases. It is observable as deference and as organisational design, although it has not been shown that individual officials lost the skill to assess cases.

What can be influenced. Almost everything: each step of the harm path is a design decision.

Nordvik Region and Hansa Mutual

Nordvik Region's benefits-triage system is exactly the kind of system these cases involved, and the audit committee asked whether the Region could become the next SyRI. The answer rests on artefacts it already had, AIDR-0007, AIDR-0011, FRIA-2026-002 and AIDR-0018, and on the one link the harm-path review found missing. The Region will publish the review's summary with the register entry, because an authority must be able to show its evidence to a court (178, Nordqvist, 2026).

Hansa Mutual faces analogous risks under insurance law. Its underwriting-support model for life and health insurance is high-risk under Annex III point 5(c), and Article 27 extends the FRIA duty to deployers of such systems (1, European Union, 2024). The group registers THR-HM-05, unfair outcomes from the model, and THR-HM-12, unlawful discrimination by it. Although its customers can go elsewhere, the four mechanisms still apply, so its review runs per market and uses complaints to national insurance ombudsmen and supervisors as its denial signals.

Summary

One class of AI dystopia has already happened without advanced AI: SyRI, the Dutch childcare benefits scandal and Robodebt.

The cases share four mechanisms: a reversed burden of proof, opacity, scale and institutional denial. The AI Act now classifies such systems as high-risk, although a calculation like Robodebt's may fall outside it and the older protections remain essential. A harm-path review names the control that blocks each path, and Nordvik Region's found its gap exactly where the Dutch and Australian institutions failed.

T7 has reached pattern at scale and is observable, though the machines were simple and the power was eventually held to account; the institutional form of T8 is observable. Chapter 12 turns to the mechanism that made one of these cases discriminatory: how bias enters automated judgement, and how it can be detected and reduced.

Sources

  1. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (178, Nordqvist, 2026)
  2. District Court of The Hague (2020). Judgment of 5 February 2020, NJCM et al. v. the State of the Netherlands (SyRI), case C/09/550982 / HA ZA 18-388, ECLI:NL:RBDHA:2020:1878. (24, District Court of The Hague, 2020)
  3. Tweede Kamer der Staten-Generaal, Parlementaire ondervragingscommissie Kinderopvangtoeslag (2020). Ongekend onrecht. (25, Tweede Kamer, 2020)
  4. Amnesty International (2021). Xenophobic Machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal. (31, Amnesty International, 2021)
  5. Royal Commission into the Robodebt Scheme (2023). Report. (26, Royal Commission, 2023)
  6. Eubanks, V. (2018). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. (169, Eubanks, 2018)
  7. Bainbridge, L. (1983). Ironies of automation. (164, Bainbridge, 1983)
  8. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
  9. European Parliament and Council of the European Union (2026). Regulation (EU) 2026/1744 (Digital Omnibus on AI). (2, European Union, 2026)
  10. European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 (General Data Protection Regulation). (6, European Union, 2016)
  11. Council of Europe (2024). Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law (CETS No. 225). (15, Council of Europe, 2024)
  12. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (179, Nordqvist, 2026)
  13. O'Neil, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. (168, O'Neil, 2016)
  14. Narayanan, A. & Kapoor, S. (2024). AI Snake Oil: What Artificial Intelligence Can Do, What It Can't, and How to Tell the Difference. (61, Narayanan & Kapoor, 2024)
  15. Acemoglu, D. & Johnson, S. (2023). Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity. (67, Acemoglu & Johnson, 2023)
  16. Narayanan, A. & Kapoor, S. (2025). AI as Normal Technology. (60, Narayanan & Kapoor, 2025)
  17. Orwell, G. (1949). Nineteen Eighty-Four. (37, Orwell, 1949)