Surveillance, Profiling and the Erosion of Privacy

The watching state is the oldest political fear about machines. In Nineteen Eighty-Four the telescreen both broadcasts and observes, and nobody knows when they are being watched (37, Orwell, 1949). Orwell's Party still needed human informers and clerks to turn observation into judgement, and that labour limited surveillance for most of the twentieth century. AI removes much of that cost, because a camera network, a mailbox or a payment ledger can now be read, linked and scored by software almost for free.

This chapter tests T7 The watching state, which Chapter 2 widened beyond the state following Shoshana Zuboff's argument that the defining surveillance of this century is commercial (170, Zuboff, 2019). Few European organisations set out to build a watching machine, but many build pieces of one: an inventory that keeps everything, a security platform that reads every message, an analytics feature that links data collected for one purpose to decisions taken for another.

Figure 13.1: Uses of AI that observe or judge people run from permitted under conditions through high-risk to prohibited under Article 5 of the AI Act; the GDPR applies across the scale.

Table 13.1: Uses along the scale.

Class	Uses
Permitted under conditions	Security monitoring of systems (works councils and unions engaged for staff); identity verification with consent; aggregate analytics without profiling
High-risk under Annex III	Post remote biometric identification; monitoring and evaluating workers (works councils and unions engaged); eligibility for public benefits; emotion recognition outside work and school
Prohibited under Article 5	Social scoring with detrimental treatment; untargeted scraping for face databases; emotion recognition at work and in education; biometric categorisation by sensitive traits; real-time remote biometric identification by police in public (narrow law enforcement exceptions with prior authorisation)

Surveillance capitalism and the state

The watching machine has two faces that share infrastructure. Zuboff describes surveillance capitalism as an economic logic that treats human experience as free raw material: behaviour is captured as data, the surplus not needed to improve a service is turned into predictions of what people will do, and those predictions are sold to those who want to influence it, through convenience rather than coercion (170, Zuboff, 2019). For states, AI changes the cost of turning collected data into knowledge about individuals. Daron Acemoglu and Simon Johnson argue that the direction of AI is a choice, repeatedly steered towards automation and surveillance rather than tools that augment workers and citizens, and treat mass monitoring of citizens in China as the clearest example of the state-led path (67, Acemoglu & Johnson, 2023).

The form most relevant to European organisations is profiling inside lawful administration: SyRI (24, District Court of The Hague, 2020) and the Dutch childcare benefits scandal (25, Tweede Kamer, 2020), in which a risk model used nationality as a risk factor (31, Amnesty International, 2021), told in Chapter 11, and the digital poorhouse Virginia Eubanks describes in the United States (169, Eubanks, 2018). None used advanced AI, which answers the open question Q11: the watching and judging state of T7 has already appeared in Europe through ordinary automation.

What AI adds is reach, in three ways. First, language models and computer vision analyse unstructured data, such as case notes, emails, recorded calls and video, for all streams at once where each once needed a human analyst. Second, they infer: Laura Weidinger and colleagues list among the risks of language models the ability to compromise privacy by correctly inferring private information that was never disclosed (70, Weidinger et al., 2022), and the International AI Safety Report 2026 notes that AI systems that analyse vast data streams could enable surveillance and social control (50, Bengio et al., 2026). Third, they make linkage cheaper: each dataset may have been collected lawfully, but combining them produces a profile no single purpose justified. SyRI was in essence a linkage machine, and cheaper linkage is done more often.

Facial recognition and scraping

Facial recognition is where T7 is most concrete, because a face cannot be left at home, and where European enforcement has gone furthest. Clearview AI built a facial-recognition search engine from photographs collected from publicly accessible websites and social media without the knowledge of the people pictured, letting customers upload a photograph and find matches. In October 2022 the French data protection authority, the Commission nationale de l'informatique et des libertés (CNIL), imposed a penalty of 20 million euros (28, CNIL, 2022). It found no legal basis under the GDPR for processing the biometric data of people in France, failures to respect rights of access and erasure, and a failure to cooperate, and ordered Clearview to stop and to delete the data, under a periodic penalty for delay.

The case matters beyond facial recognition. Publicly accessible is not the same as freely usable: Article 9 of the GDPR treats biometric data processed to identify a person uniquely as a special category whose processing is prohibited unless one of a few conditions applies (6, European Union, 2016). And the harm is structural, since a database of faces linked to identities makes anonymity in public a matter of someone else's choice, with misidentification falling hardest on the groups for whom, as Chapter 12 shows, the technology works least well. The AI Act responded directly: Article 5(1)(e) prohibits AI systems that create or expand facial-recognition databases through the untargeted scraping of facial images from the internet or CCTV footage, whatever the purpose (1, European Union, 2024).

Censorship built into models

Surveillance watches what people say; censorship decides what they may hear, and with general-purpose models the values of a developer, and of its state, travel with the weights. Jennifer Pan and Xu Xu compared how language models developed in China and elsewhere answer questions on topics politically sensitive in China (131, Pan & Xu, 2026). The models from China refused such questions more often and, when they answered, gave shorter responses and more often responses that were completely inaccurate, whether by refuting the question, avoiding the key information or fabricating. The Center for AI Standards and Innovation at the US National Institute of Standards and Technology (NIST) reported in 2025 that DeepSeek models echoed inaccurate and misleading narratives of the Chinese Communist Party markedly more often than US reference models (130, NIST, 2025); the evaluator is a government body in a period of strategic competition, although its measurements are consistent with the academic study.

Proportion matters, because every developer shapes what its models will say; the distinction lies in whose policy it is, whether it is published, and whether it serves users' safety or a state's political interests. An open-weight model deployed by a European public authority carries its training with it, so residents may receive another state's censorship through their own authority's website. AI Governance, Chapter 6, concluded that model origin is a risk question separate from hosting (178, Nordqvist, 2026); this chapter builds on that conclusion (F29) by adding political filtering to the origin assessment, tested against a reference record whenever the model changes.

Article 5 prohibitions

Article 5 of the AI Act lists practices prohibited outright (1, European Union, 2024). The prohibitions have applied since 2 February 2025, and breaches attract fines of up to 35 million euros or 7 per cent of total worldwide annual turnover, whichever is higher (Article 99(3)). Chapter 8 discussed manipulation and the exploitation of vulnerability; five others concern surveillance and profiling. What follows summarises the text of the Regulation, to be read alongside the Commission's guidelines on prohibited practices, published in February 2025 and adopted in July 2025 (184, European Commission, 2025), and is not legal advice.

Social scoring

Article 5(1)(c) prohibits AI systems that evaluate or classify natural persons or groups over a certain period of time on the basis of their social behaviour or known, inferred or predicted personal or personality characteristics, where the resulting social score leads to detrimental or unfavourable treatment either in social contexts unrelated to those in which the data was originally generated or collected, or that is unjustified or disproportionate to the behaviour or its gravity. It applies to private as well as public actors.

The first limb prohibits context transfer; the second is a proportionality test. Neither prohibits scoring as such; credit scoring, fraud detection and benefit ranking are not social scoring in themselves. The risk therefore lies in drift, since a lawful ranking model to which someone adds a feature from an unrelated context can cross from Annex III into Article 5 in one release.

Untargeted scraping and biometric categorisation

Article 5(1)(e), described above, prohibits building facial-recognition databases by untargeted scraping. Article 5(1)(g) prohibits biometric categorisation systems that deduce or infer race, political opinions, trade-union membership, religious or philosophical beliefs, sex life or sexual orientation, except for labelling or filtering lawfully acquired biometric datasets and certain law-enforcement categorisation. Other biometric categorisation is high-risk under Annex III, point 1, and Article 50(3) requires deployers to inform the people exposed to it.

Emotion recognition at work and in education

Article 5(1)(f) prohibits AI systems that infer the emotions of a natural person in the workplace and education institutions, except for medical or safety reasons; Chapter 7 applied it to schools. Under Article 3(39), an emotion recognition system infers emotions or intentions from biometric data, so inferring stress from a worker's voice or face is within it, whereas sentiment analysis of written text is not, although it may still be unlawful under the GDPR or labour law. Outside workplaces and education, emotion recognition is high-risk rather than prohibited. Moreover, Arvind Narayanan and Sayash Kapoor argue that many products claiming to read character, honesty or suitability from faces and voices do not work as claimed (61, Narayanan & Kapoor, 2024), so such systems produce surveillance and error at once.

Real-time remote biometric identification

Article 5(1)(h) prohibits real-time remote biometric identification in publicly accessible spaces for law enforcement, unless strictly necessary for one of three objectives: the targeted search for specific victims of abduction, trafficking in human beings or sexual exploitation, and the search for missing persons; the prevention of a specific, substantial and imminent threat to life or physical safety, or of a genuine and present or foreseeable threat of a terrorist attack; or the localisation or identification of a person suspected of a serious criminal offence listed in Annex II and punishable by a custodial sentence of a maximum of at least four years. Article 5(2) and (3) confine each use to the specifically targeted individual and require a fundamental rights impact assessment, registration and prior authorisation by a judicial or independent administrative authority, with urgent use allowed only if authorisation is sought within 24 hours. Under Article 5(5) each member state decides by national law whether to permit such use at all.

The prohibition applies only to law enforcement, so facial recognition by a shop, a stadium or an employer is high-risk and constrained by Article 9 of the GDPR rather than prohibited, and only to real-time use; retrospective identification is high-risk, with added safeguards under Article 26(10). Its exceptions are a political compromise between advocates of a full ban and member states that wanted wider powers.

What the Act allows

Most surveillance-adjacent AI is therefore allowed under conditions. Annex III makes high-risk the use of AI for biometric identification, categorisation and emotion recognition outside the prohibitions, for recruitment, promotion, termination, task allocation and the monitoring and evaluation of workers (point 4), and for eligibility for essential public assistance benefits (point 5(a)). The GDPR applies throughout: purpose limitation and data minimisation (Article 5(1)), data protection by design (Article 25), limits on solely automated decisions (Article 22) and impact assessments for high-risk processing such as profiling-based decisions and large-scale monitoring of public areas (Article 35) (6, European Union, 2016).

Worked example: A prohibited-practice check in the AI inventory pipeline

AI Governance, Chapter 8, made classification triggered by change rather than by the calendar, and Architecture as Code (Chapter 10) showed how a policy written once as code can be enforced at every change (178, Nordqvist, 2026) (177, Nordqvist, 2025). The illustrative check below applies that pattern to Article 5 at Nordvik Region whenever an AI inventory entry changes. The decision to run it is recorded as AIDR-0431, and it implements a control for THR-NR-04, harm through the benefits-triage system to residents' rights.

prohibited_practice_check:
  id: PPC-NR-01
  status: illustrative
  decision_record: AIDR-0431
  threat_register_entry: THR-NR-04
  owner: AI governance lead
  runs_on: [new inventory entry, change to features or data sources, change of purpose]
  rules:
    - id: ART5-1C
      practice: social scoring
      applies_if_all: [evaluates_natural_persons, produces_score_or_class, affects_treatment]
      review_if_any: [data_from_unrelated_context, features_on_social_behaviour]
      outcome: legal_review_before_merge
    - id: ART5-1E
      practice: untargeted scraping for facial-recognition databases
      applies_if_any: [scraped_facial_images, builds_face_database]
      outcome: block
    - id: ART5-1F
      practice: emotion recognition at work or in education
      applies_if_all: [infers_emotion_or_intention, biometric_input]
      settings: [workplace, education]
      exception: documented medical or safety reason
      outcome: block_unless_exception_approved
    - id: ART5-1H
      practice: real-time remote biometric identification
      applies_if_all: [biometric_input, real_time, public_space]
      outcome: block
  example_run:
    entry: benefits-triage claims ranker, proposed version 4.1.0
    change: add public-transport travel-card activity as a feature
    rule_hit: ART5-1C
    reason: data generated in an unrelated context used to rank applicants
    outcome: legal_review_before_merge

The check does not decide legality. It makes the questions of Article 5 unavoidable at the moment a change is proposed, records the answer beside the change, and fails closed: an entry that does not declare the attributes the rules read cannot be merged. In the example run nobody intended social scoring; an analyst noticed that travel-card activity improved the ranking, which is how a lawful ranking becomes a prohibited one.

The employee and the SOC

The watching machine most European organisations actually operate points inward, at their own staff, through productivity analytics, insider-risk tools and the security operations centre (SOC). The security case is the hardest, because the purpose is legitimate and the series has recommended the tools: AI Platform and Security, Chapter 18, concluded that defenders need detection at machine speed (179, Nordqvist, 2026), and AI Red Team, Chapter 18, puts AI to work in the SOC to enrich, correlate and summarise alerts (180, Nordqvist, 2026). Behind each alert is a person, however. An investigation agent that can read a mailbox to triage a phishing report can read it for other reasons, and a behavioural model that flags unusual access builds, over time, a profile of how each employee works. The difference between defending the organisation and monitoring its people lies in purpose limits, access rules and oversight, not in the technology.

European law gives employees several layers of protection. Article 88 of the GDPR allows member states to adopt more specific rules on processing in employment, including safeguards for monitoring at the workplace (6, European Union, 2016). Article 26(7) of the AI Act requires employers to inform workers' representatives and affected workers before putting a high-risk AI system into service at the workplace, and Article 2(11) preserves more favourable national and Union law (1, European Union, 2024). National labour law adds the strongest layer. In Germany, works councils co-determine the introduction of technical devices suitable for monitoring employees' behaviour or performance, which courts read broadly enough to include most security tools that log user activity; in the Netherlands a works council's consent is required for systems that make it possible to monitor staff; and in Sweden employers must negotiate with the unions before significant changes, which in practice include new monitoring tools.

These rights are better treated as a design input than as a delay. A works-council agreement states the purpose, data, retention and who may look at individual data, which are the controls a well-designed SOC needs anyway: pseudonymisation until an investigation is opened, a logged and approved step to de-pseudonymise, and a ban on using security telemetry for performance management. Because a SOC tool that inferred stress from voices would fall within Article 5(1)(f), and one used to evaluate workers is high-risk under Annex III, point 4, security AI belongs in the same inventory pipeline as every other system.

The debate

The case for concern

The case for concern begins with cost. Surveillance used to be limited less by law than by labour: the number of people who could be followed, the conversations transcribed, the files read. Commercial actors already capture behaviour at scale (170, Zuboff, 2019), and AI removes the remaining bottleneck of analysis. The only remaining limits are legal and political, and those are easier to change than physics.

Infrastructure also persists. Cameras, data lakes and linkage agreements built for one purpose are rarely dismantled; they are repurposed. Acemoglu and Johnson argue that the use of AI for surveillance is a choice powerful actors have repeatedly made because it serves their interests (67, Acemoglu & Johnson, 2023), so a system that makes total observation possible makes it available to whoever holds power next.

The European record adds weight: SyRI, the childcare benefits scandal (25, Tweede Kamer, 2020), Clearview (28, CNIL, 2022) and censorship in downloadable models (131, Pan & Xu, 2026) were each stopped or exposed only after harm was done. On this view Article 5 is necessary but fragile: it contains law-enforcement exceptions, leaves member states room to authorise real-time identification, and leaves high-risk monitoring to documentation duties rather than limits.

The case for calm

The case for calm reads the same record as democracy working, imperfectly and late. A Dutch court stopped SyRI on human-rights grounds (24, District Court of The Hague, 2020), a parliamentary inquiry exposed the childcare benefits scandal and the government resigned, and the CNIL fined Clearview and ordered deletion (28, CNIL, 2022). Europe has also made the decisive political choice: the GDPR (6, European Union, 2016) and Article 5 of the AI Act (1, European Union, 2024) prohibit social scoring, untargeted face scraping, emotion recognition at work and in schools and biometric categorisation by sensitive traits, and constrain real-time identification by police more tightly than most democracies outside Europe, and the Council of Europe convention extends the same principles to privacy and data protection (15, Council of Europe, 2024).

Capability is also overstated. Narayanan and Kapoor argue that much predictive AI about people does not work as claimed (61, Narayanan & Kapoor, 2024), and that AI is a normal technology whose effects depend on slow institutional adoption (60, Narayanan & Kapoor, 2025): surveillance capability has to be procured, integrated, staffed and justified to budget holders and courts.

Where the positions meet

Both positions accept that AI lowers the cost of observation, that profiling has caused serious harm in Europe without advanced AI, and that the corrections came from courts, parliaments and regulators. They agree on prohibitions on the most dangerous practices, independent oversight with resources, and data minimisation, because data never collected cannot be repurposed. They differ in their confidence that these safeguards will hold under an emergency or a change of government, which argues for safeguards that are hard to reverse.

What can be done

Individuals can use the GDPR's rights of access, erasure and objection (Articles 15, 17 and 21), the means by which the Clearview case reached the CNIL. Employees can ask their works council or union what monitoring agreements exist.

Organisations can collect only what a purpose needs, delete it on schedule, run a prohibited-practice check at every change, treat employee-representation agreements as design inputs for any tool that observes staff, and test models that answer questions of fact for political filtering. Public authorities can also keep a reasons log for every case and generate the algorithm register from version-controlled entries (AI Governance, Chapter 17) (178, Nordqvist, 2026).

Society can fund regulators with technical staff able to examine biometric and profiling systems, and parliaments using the option in Article 5(5) can require public reporting on every authorisation, extending the annual reporting that Article 5(6) and (7) already provide for.

Verdict on the trope

T7 The watching state. In its commercial form, the capture and analysis of behavioural data to predict and influence people at scale, the trope reaches pattern at scale as the business model of a large part of the digital economy (170, Zuboff, 2019); in its state form outside democratic constraints it does too. In Europe, profiling and surveillance that escape accountability have reached incident in the wild: SyRI, the childcare benefits scandal and Clearview are documented cases, each eventually checked by a court, a parliament or a regulator. Political filtering in models of foreign origin reaches laboratory demonstration in systematic evaluations of deployed models (131, Pan & Xu, 2026) (130, NIST, 2025), with its effect on real users not yet measured. All of these are observable. That falling costs will push organisations towards linkage, inference and monitoring they would not previously have attempted is expected. An Orwellian state in a European democracy, in which AI-enabled surveillance escapes legal accountability altogether, is speculation, and depends on political choices rather than technology.

What can be influenced. Organisations decide what data they collect and keep, whether they link it across purposes, whether every system passes a prohibited-practice check, whether staff monitoring is agreed with employee representatives, and which models may answer questions of fact. The strength of the prohibitions and of the bodies that enforce them is within the reach of citizens and their representatives.

Nordvik Region and Hansa Mutual

For Nordvik Region the risk is drift towards social scoring. The benefits-triage system ranks about 40,000 applications a month, and the Region holds data about many of the same residents in health and social-care coordination and public transport; using it to rank benefit applicants would move data from an unrelated context into a detrimental decision, the first limb of Article 5(1)(c). The Region records the risk under THR-NR-04 in ATA-NR-2026-01 and runs the prohibited-practice check on every inventory change, with the earlier removal of postcode features (AIDR-0018) as precedent. It also screens the open-weight models behind its citizen-service assistant for political filtering, and its SOC triage assistant (NR-SEC-001) is introduced only after negotiation with the unions.

Hansa Mutual's exposure is its own security monitoring. The SOC investigation agent (HM-SEC-001) serves a group SOC that runs from Hamburg and the Polish service centre, and alerts concern employees in all seven markets. Before the agent goes live in an entity, the employee-representation process must be completed, whether works-council co-determination in Germany, consent in the Netherlands or union negotiation in Sweden, using the rollout gate per entity that AI Governance, Chapter 12, established (178, Nordqvist, 2026). A group baseline agreement covers purpose, retention and pseudonymisation, and the UK and Swiss entities apply the Article 5 check as group policy. Nordvik must make sure its systems do not come to watch and judge residents, whereas Hansa must make sure the tools that protect the group do not come to watch its staff.

Summary

The watching state's traditional limit was the cost of human watchers, and AI removes much of it through analysis, inference and linkage. In Europe its state face has appeared in profiling without any advanced AI, the CNIL's penalty against Clearview AI established that scraped faces are not free raw material, and models carry the political filtering of their developers' jurisdictions.

Article 5 prohibits the most dangerous practices, while most surveillance-adjacent AI is allowed under conditions. Organisations most often meet the trope through drift and staff monitoring, where works-council and union rights are best treated as design inputs.

T7 reaches pattern at scale in its commercial and authoritarian forms and incident in the wild in Europe, where courts and regulators have so far checked it; further drift is expected, and an unaccountable watching state in a European democracy remains speculation. Chapter 14 turns from being watched at work to the work itself: what AI does to jobs, skills and the ironies of automation.

Sources

1. Orwell, G. (1949). Nineteen Eighty-Four. (37, Orwell, 1949)
2. Zuboff, S. (2019). The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. (170, Zuboff, 2019)
3. Acemoglu, D. & Johnson, S. (2023). Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity. (67, Acemoglu & Johnson, 2023)
4. District Court of The Hague (2020). Judgment of 5 February 2020, NJCM et al. v. the State of the Netherlands (SyRI). (24, District Court of The Hague, 2020)
5. Tweede Kamer der Staten-Generaal (2020). Ongekend onrecht (Unprecedented injustice). (25, Tweede Kamer, 2020)
6. Amnesty International (2021). Xenophobic Machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal. (31, Amnesty International, 2021)
7. Eubanks, V. (2018). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. (169, Eubanks, 2018)
8. Weidinger, L., Uesato, J., Rauh, M., Griffin, C. et al. (2022). Taxonomy of Risks posed by Language Models. (70, Weidinger et al., 2022)
9. Bengio, Y. et al. (2026). International AI Safety Report 2026. (50, Bengio et al., 2026)
10. Commission nationale de l'informatique et des libertés (2022). Facial recognition: 20 million euros penalty against CLEARVIEW AI. (28, CNIL, 2022)
11. European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 (General Data Protection Regulation). (6, European Union, 2016)
12. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act). (1, European Union, 2024)
13. Pan, J. & Xu, X. (2026). Political censorship in large language models originating from China. (131, Pan & Xu, 2026)
14. National Institute of Standards and Technology, Center for AI Standards and Innovation (2025). CAISI Evaluation of DeepSeek AI Models Finds Shortcomings and Risks. (130, NIST, 2025)
15. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (178, Nordqvist, 2026)
16. Narayanan, A. & Kapoor, S. (2024). AI Snake Oil: What Artificial Intelligence Can Do, What It Can't, and How to Tell the Difference. (61, Narayanan & Kapoor, 2024)
17. Council of Europe (2024). Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law (CETS No. 225). (15, Council of Europe, 2024)
18. Nordqvist, G. (2025). Architecture as Code. (177, Nordqvist, 2025)
19. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (179, Nordqvist, 2026)
20. Nordqvist, G. (2026). AI Red Team: AI as an Active Part of Cyber Defence in European Enterprises and Public Authorities. (180, Nordqvist, 2026)
21. Narayanan, A. & Kapoor, S. (2025). AI as Normal Technology. (60, Narayanan & Kapoor, 2025)
22. European Commission (2025). Commission Guidelines on prohibited artificial intelligence practices established by Regulation (EU) 2024/1689 (AI Act). (184, European Commission, 2025)