Discrimination at Scale: Bias in Automated Judgement

Chapter 11 showed that the watching and judging state of science fiction has already appeared in European public administration without any advanced AI. This chapter follows one strand of those failures beyond the public sector: discrimination. When a system that ranks, scores or classifies people works systematically worse for some groups, the harm is no longer one unfair decision but the same unfair decision repeated for every person who passes through the system, with an appearance of neutrality that a human decision-maker rarely enjoys.

The chapter tests T7 The watching state against that evidence. The subject matters most to public authorities and insurers, whose AI systems the AI Act classifies as high-risk (1, European Union, 2024). AI Governance concluded that in public authorities legality comes before capability, and that trouble starts where statistical models replace rules without explanation discipline (178, Nordqvist, 2026); discrimination is where that trouble is most often found.

Figure 12.1: Bias can enter at every stage of the lifecycle of a system that judges people, and the feedback from retraining closes the loop.

Table 12.1: Where bias enters and the check that can detect it.

Lifecycle stage	Where bias enters	Check
Frame the decision	Problem framing: whom to scrutinise	FRIA names groups at risk
Collect and label data	Target label (cost for need); under-represented groups	Label and data review
Train the model	Features that proxy protected traits	Proxy analysis in a test enclave
Release gate		Disparate-impact test per release
Decide about people	Uneven following of scores	Monitoring and override rates; appeals and contestation
Retrain	Feedback loop: outputs become labels	Appeals and contestation

Bias in, bias out, at scale

The phrase "bias in, bias out" is accurate but incomplete. The NIST AI Risk Management Framework distinguishes systemic bias in institutions and the data they produce, computational and statistical bias from unrepresentative samples and modelling choices, and human-cognitive bias in how people design, interpret and act on a system (72, NIST, 2023). All three appear in the cases below, usually together.

Where bias enters

The first point of entry is the framing of the problem: predicting who is likely to commit benefit fraud, rather than who is likely to be underpaid, already shapes whose behaviour is scrutinised. The second is the target label. A model cannot learn "health need" directly; it learns a measurable stand-in, such as past health spending, and inherits every way in which that stand-in differs between groups. The third is representation in the data: a face-analysis system trained mostly on lighter-skinned faces, or a language model trained mostly on a few dominant languages, performs worse for those under-represented, and Laura Weidinger and colleagues placed discrimination, exclusion and toxicity first in their taxonomy of risks from language models (70, Weidinger et al., 2022).

The fourth is proxies. Removing a protected characteristic such as ethnic origin does not remove its influence, because postcode, first language, school or the length of a residence record correlate with it, and a model will use them if they help the prediction. The fifth is the use of output: staff may follow a score more readily for some applicants than for others, so that a system balanced on paper produces unbalanced decisions in practice.

Why it scales

Human bias is real but distributed: officials differ, some prejudices cancel out, and one official reaches a limited number of people. An automated system, by contrast, applies the same learned pattern to every case until someone changes it, so its unfairness is consistent, which makes it both more damaging and more detectable. Cathy O'Neil described the models she called weapons of math destruction by opacity, scale and damage to jobs, credit, liberty and housing, and added the feedback loop: a model that directs police patrols to neighbourhoods with more recorded crime produces more records there, which confirms the model (168, O'Neil, 2016). Virginia Eubanks added the institutional dimension, arguing from automated eligibility, child-welfare and homelessness systems in the United States that such systems are deployed first and most intensively on poor and marginalised people, who have least power to contest the result (169, Eubanks, 2018). This is where bias meets T7: the watching is uneven, and so is the judging.

Documented cases

Recognition accuracy: Gender Shades

In 2018 Joy Buolamwini and Timnit Gebru evaluated three commercial facial-analysis services that classified faces by gender (102, Buolamwini & Gebru, 2018). Because existing benchmarks were dominated by lighter-skinned subjects, they built a new one of parliamentarians from three African and three European countries, balanced by gender and skin type, and reported results by their intersection. All three services performed worst on darker-skinned women, whose error rate reached about a third for the worst-performing service, against below one per cent for lighter-skinned men; two services performed best on lighter-skinned men and the third on darker-skinned men. Aggregate accuracy hid the disparity, which was largest at an intersection that a test by gender or skin type alone would have understated, in products already on the market. Chapter 13 takes up the European debate on facial recognition that followed.

Health-risk scores: the cost proxy

In 2019 Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan examined a commercial algorithm used by a large US health system to identify patients for extra care (103, Obermeyer et al., 2019); the authors cite industry estimates that such algorithms are applied to around 200 million people a year in the United States. It did not use race as an input, but predicted future health-care costs as a stand-in for future health needs. At the same risk score, Black patients were considerably sicker than white patients, because less money was spent on Black patients with the same level of need, reflecting unequal access to care. Correcting the disparity would raise the share of Black patients automatically identified for the programme from 17.7 to 46.5 per cent, and changing the label to a combination of cost and direct health measures substantially reduced the bias. The model was accurate by its own measure; the discrimination lay in the choice of what to predict.

Welfare profiling

The third family of cases is told in Chapter 11. In the Dutch childcare benefits affair, Amnesty International found that the tax authority's risk-classification model used nationality, Dutch or not, as a risk factor, and concluded that this amounted to discrimination and racial profiling (31, Amnesty International, 2021); the parliamentary inquiry called the parents' treatment unprecedented injustice (25, Tweede Kamer, 2020). The SyRI court also observed that the opacity of the model made its risk of discriminatory effects impossible to assess (24, District Court of The Hague, 2020).

What the cases have in common

In each case bias was found in systems that passed their operators' tests, it was found by outsiders, it took a different form, and the people affected could not see the cause until someone else investigated. Narayanan and Kapoor generalise the last point: predictive AI applied to decisions about people is often sold on untested accuracy claims, and the people it misjudges have little means of finding out why (61, Narayanan & Kapoor, 2024).

Fairness metrics and their trade-offs

Detecting bias requires a measurable definition of fairness, and there are several reasonable definitions that cannot in general all be satisfied at once.

The main families of metrics

The simplest family compares outcomes. Demographic parity asks whether the rate of an outcome, such as the share flagged for verification, is the same across groups; the disparate-impact ratio divides the rate for one group by the rate for the most favoured group. United States employment practice has long treated a ratio below 0.8 as evidence of adverse impact, the four-fifths rule. European law has no fixed threshold, but the ratio is a useful screening signal rather than a verdict.

The second family compares errors. Equal opportunity asks whether the true positive rate is the same across groups; equalised odds asks the same of both true and false positive rates. In benefits triage, a lower true positive rate for one group means its genuinely urgent cases are ranked too low more often; in a fraud model, a higher false positive rate means its honest members are investigated more often. The third family concerns calibration, whether a given score means the same thing for every group, and the Obermeyer case is a calibration failure measured against need.

Why no single metric settles it

Research in algorithmic fairness, much of it prompted by the dispute over a criminal-recidivism risk tool in the United States, has shown that when base rates differ between groups, an imperfect classifier cannot in general be both well calibrated within each group and have equal error rates across them. In that dispute one side showed that error rates differed between Black and white defendants, whereas the other replied that the scores were equally calibrated; both were right about what they measured.

Choosing a metric is therefore a normative decision about which errors matter more and to whom: in benefits triage an urgent case ranked low is probably the most serious error, whereas in a verification queue the burden falls on those selected. Choose the metrics before testing and record why, measure more than one, and treat them as instruments for finding problems, not as a certificate of fairness. AI Governance, Chapter 10, reaches the same conclusion (178, Nordqvist, 2026).

The legal frame

European law approaches discrimination by automated judgement through the AI Act, fundamental-rights instruments and equality law. What follows summarises their architectural implications, not legal advice.

The AI Act

Article 10 of the AI Act requires that data for high-risk AI systems be examined for possible biases likely to affect health and safety, fundamental rights or lead to prohibited discrimination, especially where outputs influence inputs for future operations, and that measures be taken to detect, prevent and mitigate them (1, European Union, 2024). The clause about outputs influencing inputs is O'Neil's feedback loop in legislative form.

Article 10(5) addresses a dilemma every bias tester meets: detecting discrimination by ethnic origin or health status requires data the General Data Protection Regulation (GDPR) treats as special categories under Article 9 (6, European Union, 2016). Providers may process such data where strictly necessary for bias detection and correction, under safeguards that include pseudonymisation, strict access controls and deletion once the bias is corrected, which implies a separate, access-controlled test enclave rather than protected attributes in the production feature store.

The fundamental rights impact assessment (FRIA) of Article 27 applies before deployment to public bodies, private entities providing public services, and deployers of systems for creditworthiness assessment and for life and health insurance pricing; a FRIA that names the groups at risk and the metrics monitored for them becomes a test plan. Article 26 requires deployers to monitor operation, and Article 86 gives a person affected by a decision based on an Annex III system the right to a clear and meaningful explanation of its role, which is the hook for contestation.

Fundamental rights and equality law

The Charter of Fundamental Rights of the European Union prohibits discrimination on grounds including sex, race, colour, ethnic or social origin, genetic features, language, religion, disability, age and sexual orientation, and EU equal-treatment directives give effect to it through national law. They distinguish direct discrimination, less favourable treatment because of a protected characteristic, from indirect discrimination, where an apparently neutral criterion puts people with a protected characteristic at a particular disadvantage unless objectively justified by a legitimate aim with appropriate and necessary means.

Using nationality as a risk factor is close to direct discrimination, whereas a proxy such as postcode is a candidate for indirect discrimination. In insurance, a 2011 judgment of the Court of Justice of the European Union established that sex may not be used as a factor resulting in differences in individual premiums and benefits, even where actuarially relevant: predictive power is not in itself a defence. Moreover, once a claimant shows facts from which discrimination may be presumed, the burden often shifts to the respondent, and indirect discrimination is established by effects, not intent, so a disparate-impact test is the kind of evidence the law expects. The Council of Europe's Framework Convention extends commitments to equality and effective remedies to signatories outside the EU, such as the United Kingdom (15, Council of Europe, 2024).

Detecting and reducing bias

The measures that work fall into three groups, testing before release, monitoring in operation and contestation by the people affected, and each catches what the others miss.

Testing before release

Testing begins with the target: what the label measures, how it was generated and whether it could differ between groups for reasons unrelated to the quantity of interest. The Obermeyer case would have been caught at this step. Proxy analysis follows: measure how strongly each feature predicts protected attributes in the test enclave, and for strong proxies require a written justification of legitimate aim and necessity, or remove the feature.

The disparate-impact test then compares the chosen metrics across groups on held-out data, handling small groups, because a ratio computed on thirty people is noise, intersections, because Gender Shades showed that the largest disparities may appear only there, and uncertainty, through confidence intervals. It runs for every new model version, following F10: testing is triggered by change, not by the calendar (178, Nordqvist, 2026). AI Platform and Security, Chapter 20, describes how such evaluation gates sit in the delivery pipeline and produce versioned evidence (179, Nordqvist, 2026). Mitigation options include changing the label, removing proxy features, reweighting or collecting more data for under-represented groups, adjusting thresholds where the law permits, and, often most effective, changing how the output is used, for example from selection for investigation to selection for assistance.

Worked example: A disparate-impact test configuration in a release pipeline

The following configuration is illustrative. It belongs to Nordvik Region's benefits-triage system, whose claims ranker is retrained quarterly by the provider inside the Region's platform, and implements a control for THR-NR-04, harm through the benefits-triage system to residents' rights. Protected attributes are joined only inside a test enclave under Article 10(5), and the gate blocks promotion when a threshold is breached.

disparate_impact_test:
  id: DIT-NR-CR-01
  status: illustrative
  system: benefits-triage claims ranker
  model_version_under_test: 4.1.0
  baseline_version: 4.0.0
  decision_records: [AIDR-0018]
  fria: FRIA-2026-002
  threat_register_entry: THR-NR-04
  run_on: [new model version, quarterly retraining, drift alert, change to FRIA]
  data:
    protected_attributes_source: test enclave, pseudonymised, Article 10(5) safeguards
    attributes: [sex, age_band, born_abroad, disability_benefit_recipient]
    intersections: [[sex, born_abroad], [sex, age_band]]
    min_group_size: 200
    confidence: bootstrap 95 per cent interval, 1000 resamples
  outcomes:
    - name: ranked_urgent
      metric: selection_rate_ratio
      block_if_lower_bound_below: 0.80
    - name: selected_for_manual_verification
      metric: selection_rate_ratio_inverse
      block_if_lower_bound_below: 0.80
    - name: urgent_case_found
      metric: true_positive_rate_gap
      block_if_gap_above: 0.05
  proxy_check:
    flag_if_attribute_predicted_with_auc_above: 0.70
    action: written justification or removal before release
  on_block:
    promote: false
    notify: [Head of Benefits Unit, AI governance lead, data protection officer]
  reporting:
    publish_summary_in: public algorithm register entry
    monitor_in_production: monthly, same metrics on decided cases

The thresholds are starting points, not legal standards: a breach does not prove discrimination but stops the release until people have looked at why. The intersections follow the FRIA's analysis of the groups most at risk, born_abroad reflects the lesson of the Dutch case, and the register summary follows F30.

Monitoring in operation

A test on historical data cannot show how a system behaves once people use it, so monitoring computes the same metrics on decided cases, tracks overrides per group and watches for drift. Overrides cut both ways: if staff override more often for one group, either they are correcting a biased system or adding bias of their own. AI Platform and Security, Chapter 21, treats drift and observability in production (179, Nordqvist, 2026). Monitoring must also watch for the feedback loop that Article 10 names: where investigation results become training labels, the model learns from its own selections, so a random sample selected independently of the model keeps a comparison the model did not choose.

Contestability

The people affected are often the first to notice that something is wrong and the last to be asked. Contestability therefore needs notice that an automated system played a part, an explanation of its role under Article 86 of the AI Act and the GDPR, and a route to a person with authority to change the outcome, with successful challenges fed back to the team that owns the model. A rising rate of successful appeals from one group is a monitoring signal as strong as any statistical metric, so logging appeals by system version closes the loop the documented cases lacked.

The debate

That automated systems can discriminate is no longer in dispute. The debate is whether automated judgement is worse than the human judgement it replaces, and whether its measurability makes it easier or harder to correct.

The case for concern

The case for concern begins with what the documented cases share: a system its operators considered adequate treated a group worse, and nobody inside noticed. Buolamwini and Gebru found error rates for darker-skinned women dozens of times higher than for lighter-skinned men in products on sale (102, Buolamwini & Gebru, 2018); Obermeyer and colleagues found an algorithm applied to millions of patients underestimating the needs of Black patients (103, Obermeyer et al., 2019); Amnesty International found a European tax authority using nationality as a risk factor (31, Amnesty International, 2021).

The problem, on this view, is structural. Automated judgement is trained on records that encode historical injustice in care, policing and lending, so an accurate model reproduces that injustice at scale and out of sight (168, O'Neil, 2016), against people with least power to resist (169, Eubanks, 2018). The concern extends to generative AI, which absorbs the stereotypes of its training data and is increasingly used to summarise applications and draft assessments (70, Weidinger et al., 2022); its bias is harder to measure, and the International AI Safety Report 2026 notes that its focus on emerging risks is narrower than that of the 2025 edition, which also addressed bias (50, Bengio et al., 2026). T7 then needs no malevolent state, only an indifferent institution with a biased model.

The case for calm

The case for calm does not deny the cases; it denies that they show automated judgement to be worse than human judgement. Research on human decisions in hiring, lending, sentencing and medicine has long documented substantial bias and inconsistency, mostly invisible because decisions cannot be rerun with one characteristic changed. Automated systems are the first form of judgement that can be audited systematically: rerun with a protected attribute changed, error rates computed for every group, inputs inspected for proxies, then corrected and retested. The Obermeyer study supports this side as well, since the bias was found, its mechanism identified and a change of label substantially reduced it (103, Obermeyer et al., 2019); the same exercise for thousands of clinicians would be practically impossible.

This position also points to a legal frame that requires bias examination and a FRIA, and a court that has already stopped a profiling system (24, District Court of The Hague, 2020) (1, European Union, 2024); as Narayanan and Kapoor argue of AI as a normal technology, institutions shape its effects over time (60, Narayanan & Kapoor, 2025).

Where the positions meet

Both positions accept that bias in automated judgement is real and documented, that human judgement is also biased, and that auditability is an advantage only if someone actually audits. Both note that the cases were found by people with access to the systems. Three conclusions follow: bias must be measured with metrics chosen in advance and reported openly; measurement requires access to the system and to protected-attribute data under safeguards, which the AI Act now permits; and affected people need a way to contest outcomes, because they detect what aggregate statistics miss. The positions differ on how often organisations will do these things without pressure, not on whether they should.

What can be done

Individuals can ask whether an automated system played a part in a decision, request an explanation under Article 86 of the AI Act or the GDPR, and appeal.

Organisations can review labels and proxies, gate every model version as in the worked example, monitor in production, keep a random comparison sample, log appeals by system version, and record the risk in the AI threat register (Chapter 32).

Society can enforce equality law, fund independent audits, give regulators and researchers access to systems and data under safeguards, and expand public algorithm registers.

Verdict on the trope

T7 The watching state. The strand of the trope this chapter tests, the machine as an unaccountable judge that treats some people worse than others, has reached pattern at scale: discriminatory outcomes from automated judgement are documented in recognition, health care and welfare, in systems applied to millions of people on more than one continent, and their mechanisms are understood. The outlook is observable. That generative AI will carry the pattern into new domains, such as screening applications and correspondence, is expected, because the mechanisms are the same and measurement is harder.

The evidence does not, however, support the trope's implication that the machine's judgement is beyond correction: the same cases show bias being found, explained and reduced once someone with access looked. The unaccountability of T7 is a property of institutions that deploy systems without measurement, explanation or remedy, not of the technology.

What can be influenced. Almost all of it. Organisations decide what to predict, which features and metrics to use, when to stop a release and whether people can contest the result; society decides whether equality law is enforced and whether independent auditors have access.

Nordvik Region and Hansa Mutual

Nordvik Region's exposure is the one this chapter describes most directly. It released version 4.0.0 of the claims ranker without postcode features (AIDR-0018), because postcode acted as a proxy for residents' origin and income and could not be justified as necessary for ranking urgency; that decision is the reference case in ATA-NR-2026-01 for a harm path closed before it caused harm, but it addressed only one proxy. Register entry THR-NR-04 now carries the disparate-impact gate, linked to FRIA-2026-002, and because testing capacity sits partly with the supplier, the gate runs in the Region's own platform and the contract obliges the provider to supply the evidence.

Hansa Mutual's underwriting-support model for life and health products is high-risk under Annex III point 5(c), and its register carries THR-HM-05, unfair outcomes from the model, and THR-HM-12, unlawful discrimination by it in one or more markets. Calibration to risk is the core of insurance pricing, yet sex may not produce differences in individual premiums in the EU, and rules on health data, disability and age differ across its seven jurisdictions, including the UK and Swiss entities outside the AI Act. Hansa therefore runs one test suite with market overlays that declare protected attributes, permitted rating factors and thresholds, and blocks release in any market whose overlay fails.

Summary

Bias enters automated judgement through the framing of the problem, the label, the data, proxies and the use of output, and because one model applies one pattern to every case, unfairness is consistent, scaled and potentially self-confirming. Gender Shades, the Obermeyer study and the Dutch childcare benefits affair each show a different mechanism, and in each outsiders found the bias. Fairness metrics cannot all be satisfied at once, so choosing among them is a normative decision made in advance.

European law expects that evidence: the AI Act requires bias examination, a FRIA and a right to explanation, while equality law judges indirect discrimination by its effects.

Discrimination by automated judgement is observable at the rung of pattern at scale, but its unaccountability is a property of institutions rather than technology. Chapter 13 turns from judgement to the watching itself: surveillance, profiling and the erosion of privacy.

Sources

1. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
2. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (178, Nordqvist, 2026)
3. National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). (72, NIST, 2023)
4. Weidinger, L., Uesato, J., Rauh, M., Griffin, C. et al. (2022). Taxonomy of Risks posed by Language Models. (70, Weidinger et al., 2022)
5. O'Neil, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. (168, O'Neil, 2016)
6. Eubanks, V. (2018). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. (169, Eubanks, 2018)
7. Buolamwini, J. & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. (102, Buolamwini & Gebru, 2018)
8. Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. (103, Obermeyer et al., 2019)
9. Amnesty International (2021). Xenophobic Machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal. (31, Amnesty International, 2021)
10. Tweede Kamer der Staten-Generaal, Parlementaire ondervragingscommissie Kinderopvangtoeslag (2020). Ongekend onrecht. (25, Tweede Kamer, 2020)
11. District Court of The Hague (2020). Judgment of 5 February 2020, NJCM et al. v. the State of the Netherlands (SyRI). (24, District Court of The Hague, 2020)
12. Narayanan, A. & Kapoor, S. (2024). AI Snake Oil: What Artificial Intelligence Can Do, What It Can't, and How to Tell the Difference. (61, Narayanan & Kapoor, 2024)
13. European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 (General Data Protection Regulation). (6, European Union, 2016)
14. Council of Europe (2024). Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law (CETS No. 225). (15, Council of Europe, 2024)
15. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (179, Nordqvist, 2026)
16. Bengio, Y. et al. (2026). International AI Safety Report 2026. (50, Bengio et al., 2026)
17. Narayanan, A. & Kapoor, S. (2025). AI as Normal Technology. (60, Narayanan & Kapoor, 2025)