The End of Truth? Synthetic Content and the Information Ecosystem

In Nineteen Eighty-Four, Winston Smith works at the Ministry of Truth, rewriting old newspaper articles so that the record always agrees with what the Party says today, while machines compose the songs and novels fed to the population (37, Orwell, 1949). The horror is not a single lie but a world in which no one can establish what happened. When Geoffrey Hinton left Google in 2023 to speak freely about AI, the first danger he was reported to name was a version of that world: a flood of synthetic text, images and video in which ordinary people would no longer be able to tell what is true (54, Metz, 2023).

This chapter tests T10 The end of truth at the scale of society. Chapter 8 tested the trope in a single conversation and Chapter 20 examines fraud and influence operations; this chapter looks at the commons between them, the shared information through which a society learns what is happening and decides. The chapter builds on F30 from AI Governance (SERIES_FOUNDATIONS.md): in public authorities legality comes before capability, and every decision needs a record that a citizen can inspect (178, Nordqvist, 2026). Applied to what an authority says, the same discipline begins an answer to T10.

Figure 10.1: Synthetic content enters the information cycle, and provenance, labelling and accountable sources restore the ability to establish what is real.

Table 10.1: Where synthetic content enters and where trust is restored.

Stage	Where synthetic content enters	Where trust is restored
Production	Slop and fabricated media	Signing at source and canonical records; AI Act Article 50 marking and labelling
Distribution	AI summaries replace visits to sources	DSA systemic risk duties for platforms
Consumption and sharing	Real evidence dismissed as fake	Provenance checks before sharing; trusted institutions and media literacy
Archives and training data	Models trained on model output	Curated human data and visible corrections

The liar's dividend

The most corrosive effect of synthetic media may not be the fakes that are believed but the real evidence that is not. Legal scholars writing about deepfakes in 2019 called it the liar's dividend: once the public knows that convincing audio and video can be fabricated, anyone caught on a genuine recording can claim it was fabricated, and some will believe the claim.

The mechanism needs no fake, only its plausible possibility. A fraudster, by contrast, needs a fake believed long enough to move money, as when a Hong Kong employee of an engineering firm transferred about 25 million US dollars after a video call in which every other participant was synthetic (153, Magramo, 2024). The International AI Safety Report 2026 counts the manipulation of information among harms already under way, while noting how hard its effect on belief is to measure (50, Bengio et al., 2026). That distinction runs through this chapter: the capability to fabricate is established, whereas its effect on belief and trust is far less certain.

Slop and the polluted commons

The word that came into general use in 2024 for low-quality synthetic content produced at scale is "slop": generated websites that collect advertising revenue, imitation books, improbable images on social feeds, reviews written by no customer. Its defining property is indifference to truth, and it is economic before it is technical, because generative models have cut the marginal cost of plausible content to almost nothing while advertising and search still pay for attention. The harm is cumulative, since millions of generated articles make search results less useful and every unfamiliar website suspect. Emily Bender, Timnit Gebru and colleagues warned in 2021 that readers attribute meaning and intent to fluent model output that has neither behind it, and that bad actors can use it to produce large quantities of seemingly coherent text with no investment in its truth (94, Bender et al., 2021). Reliable measurement of the share of synthetic content on the web is scarce; NIST's profile for generative AI nevertheless lists information integrity among the risks particular to the technology and recommends provenance tracking and content authentication (73, NIST, 2024).

Model collapse

Slop also affects the models, because future models will learn partly from the output of earlier ones. Ilia Shumailov and colleagues trained successive generations of models, each on data produced by the previous one, and found a degenerative process they called model collapse (97, Shumailov et al., 2024). Early on, the rare events in the original data, the tails of the distribution, disappear; later, the generations converge on a narrow distribution that bears little resemblance to the original. Keeping a share of the original human-written data in each generation reduced the effect.

For the commons, the tails that collapse first are the unusual, the minority, the local and the specialist. However, the real web accumulates synthetic content alongside human content and developers curate what they use, which researchers have argued softens the effect considerably. The result therefore demonstrates a mechanism under defined conditions rather than forecasting degradation, and Chapter 16 takes up whether a culture of summaries of summaries loses its tails in the same way.

Search, summaries and the economics of news

Facts a society can rely on are produced, mostly by journalists, researchers, statisticians and public authorities, and for two decades much of the money for news has depended on readers arriving through search engines and social platforms. The Pew Research Center measured how generative AI changes that route from the browsing data of 900 adults in the United States in March 2025 (101, Chapekis & Lieb, 2025). When a Google search produced an AI summary, users clicked on a link to another website in about 8 per cent of visits, against about 15 per cent without one, and on a source cited inside the summary in about 1 per cent.

The Reuters Institute's Digital News Report 2025, surveying about 100,000 people in 48 markets, found trust in news stable at about 40 per cent, among the highest in Finland (100, Newman et al., 2025). About 7 per cent of respondents, and about 15 per cent of those under 25, used AI chatbots for news each week, and most worried about telling real from fake online.

Fewer readers reaching the source means less income, especially for local and specialist outlets. Moreover, generated summaries are subject to hallucination (95, Ji et al., 2023), so a fluent, unattributed and occasionally wrong answer replaces a link to a publisher who can be held to account; the summary needs the journalism and reduces the income that pays for it. The Directive on copyright in the Digital Single Market grants press publishers a right over online use of their publications in Article 15, and in Article 4 lets rightsholders reserve their works from commercial text and data mining in a machine-readable way (8, European Union, 2019). For any publishing organisation, the implication is that the canonical version of what it says should live on a channel it controls.

False news spreads faster

The fear that AI will end truth assumes the information ecosystem worked tolerably before 2022. Soroush Vosoughi, Deb Roy and Sinan Aral studied about 126,000 rumour cascades on Twitter between 2006 and 2017, spread by about three million people and classified by six independent fact-checking organisations (99, Vosoughi et al., 2018). False news spread farther, faster, deeper and more broadly than true news in every category, most strongly for political news. The most widely shared false stories routinely reached between 1,000 and 100,000 people, whereas true stories rarely reached more than 1,000, and false stories were about 70 per cent more likely to be shared. Automated accounts spread true and false news at about the same rate, so human behaviour drove the difference, and the authors suggested that novelty explains part of the appeal.

It follows that the bottleneck in the spread of falsehood was never production but whether people found a story worth sharing. The dynamic arises from human attention meeting ranking systems that reward engagement, which partly answers this book's open question Q11: it was established through ordinary automation. Cheap synthetic content therefore matters most by making false stories more novel, more tailored and harder to check.

The European case that drew most attention was electoral. On 6 December 2024 the Constitutional Court of Romania annulled the presidential election process after the first round, citing irregularities that included the non-transparent use of digital technologies and artificial intelligence and undeclared campaign financing (27, Constitutional Court of Romania, 2024). The decision, without precedent in the Union and still contested, rested on reports of coordinated promotion on social media rather than on a showcase deepfake.

Labelling, provenance and platform duties

European law responds at several points of the information cycle. Beneath it, Article 5 of the Council of Europe's Framework Convention on AI requires parties to adopt measures so that AI systems are not used to undermine the integrity of democratic processes, including fair access to public debate and the ability to form opinions freely (15, Council of Europe, 2024).

Article 50 of the AI Act

Article 50 of the AI Act sets transparency duties at two points (1, European Union, 2024). Under Article 50(2), providers of systems, including general-purpose AI systems, that generate synthetic audio, image, video or text must mark their output in a machine-readable format so that it is detectable as artificially generated or manipulated, as far as technically feasible. Under Article 50(4), deployers who generate or manipulate image, audio or video content that constitutes a deepfake must disclose it, with a lighter duty for evidently artistic, satirical or fictional work; deployers who publish AI-generated text to inform the public on matters of public interest must disclose that too, unless the text has undergone human review or editorial control and a person holds editorial responsibility for it. Both the provider's marking and the deployer's labelling therefore belong in the publishing chain, and the exception puts a premium on recorded human review.

The transparency obligations were set to apply from 2 August 2026; however, the Digital Omnibus on AI amended parts of the implementation timetable, so organisations should check the dates that apply to their systems (2, European Union, 2026). AI systems intended to influence the outcome of an election or people's voting behaviour are high-risk under Annex III, and the General-Purpose AI Code of Practice treats harmful manipulation among the systemic risks that providers of the most capable models must assess (3, European Commission, 2025).

The Code of Practice and the Digital Services Act

In June 2026, under Article 50(7), the Commission published the Code of Practice on marking and labelling AI-generated content (5, European Commission, 2026). It is voluntary, but adherence is a recognised way of demonstrating compliance. It asks providers to combine machine-readable techniques rather than rely on one, because each is weak alone: metadata is easily stripped, and many platforms remove it on upload; watermarks can be weakened by editing, and free-form text cannot carry metadata and its watermarks are less reliable; detection classifiers err both ways and lose accuracy as generators improve.

The Digital Services Act binds the platforms through which most content reaches people (7, European Union, 2022). Very large online platforms and search engines must assess systemic risks under Article 34, including effects on civic discourse and electoral processes, and mitigate them under Article 35; Article 35(1)(k) names prominent markings for generated or manipulated content that would falsely appear authentic. Article 40 gives vetted researchers access to platform data, and the notice-and-action mechanism of Article 16 and the trusted flaggers of Article 22 give organisations a route to have impersonating content removed.

What labels and provenance can and cannot do

A label says that content was generated or manipulated; provenance says where it came from and what happened to it since. Labels help against the fake that would otherwise be believed, whereas provenance helps against the liar's dividend, because a liar can always claim that a genuine recording is a fake whose label was stripped. The stronger defence is authentic content carrying a verifiable signature from its publisher. The open content-credentials standard of the Coalition for Content Provenance and Authenticity (C2PA) binds cryptographically signed statements about origin and edits to a file; since a platform can strip them, a signed file should be backed by a canonical copy, with its hash, on a channel the publisher controls.

Practical guide: Publishing with provenance

Purpose. Make an organisation's own statements verifiable, label its AI-generated content in line with Article 50 and the Code of Practice, and reduce its exposure to impersonation and the liar's dividend.

Roles. The head of communications owns the policy, the chief information security officer the signing keys and impersonation monitoring, and the AI governance lead the link to the AI threat register.

Steps.

1. Mark one channel as canonical for official statements. Default: a statements page on the organisation's own domain.
2. Classify content by AI involvement: none, AI-assisted with human editorial responsibility, and AI-generated or manipulated media. Default: synthetic media depicting a real person, place or event is a deepfake for labelling purposes.
3. Label as the law and the code require, and record the named editor for AI-assisted public-interest text.
4. Sign at source with content credentials, keeping keys in a hardware security module rotated annually.
5. Publish the canonical record: text, date, signed media and a SHA-256 hash of each file, within 30 minutes of publication elsewhere.
6. Correct visibly in a public log; never silently edit a published statement.
7. Prepare for the fake and the false denial. Name who confirms or denies a recording attributed to the organisation. Default: a first statement within two hours in office hours, four outside them.
8. Monitor impersonation and report it through DSA notices or trusted flaggers.
9. Test the chain twice a year: check that credentials survive each channel, hashes match and staff follow the confirmation procedure.

Definition of done. The policy is in version control, the canonical channel publishes hashes and signed media, synthetic media carry labels, a corrections log is public, the confirmation procedure has been rehearsed, and the measures are linked to an owned entry in the AI threat register.

Worked example: A content-provenance policy for a public authority's communications

The following policy is illustrative. It belongs to Nordvik Region, implements the controls for THR-NR-08 (loss of trust in the Region's public information through synthetic content) and THR-NR-03 (synthetic-media impersonation of elected officials and staff), is recorded as AIDR-0420, and lives in version control beside the AI threat register.

content_provenance_policy:
  id: CPP-NR-2026-01
  status: illustrative
  decision_record: AIDR-0420
  threat_register_entries: [THR-NR-08, THR-NR-03]
  owner: Head of Communications
  key_custodian: CISO
  canonical_channel:
    url_path: /statements
    publishes: [statement_text, publication_time, signed_media, sha256_hashes]
    max_delay_minutes: 30
  content_classes:
    - class: ai_assisted_text
      label: none
      condition: named editor recorded under Article 50(4) editorial responsibility
    - class: synthetic_media
      label: visible label per Code of Practice on marking and labelling
      depicts_real_person_or_event: treat as deepfake
      prohibited_uses: [synthetic voice or likeness of elected officials]
  signing:
    standard: C2PA content credentials
    applies_to: [image, video, audio, pdf]
    key_storage: hardware security module
    key_rotation_months: 12
  corrections:
    log: public
    silent_edits: prohibited
  authenticity_queries:
    confirm_or_deny_owner: Head of Communications with CISO
    first_response_hours: {office_hours: 2, out_of_hours: 4}
  impersonation_monitoring:
    reporting_route: DSA Article 16 notice, trusted flagger where available
  tests:
    cadence_months: 6
    checks: [credentials survive each channel, hash matches canonical copy, confirmation drill]

The defaults are the Region's choices, not requirements of law.

The debate

The case for concern

The case for concern starts from cost. For most of history a convincing photograph, recording or document took skill, time and money, and that cost was a guarantee: most of what one saw was real because faking it was hard. Generative models have removed the guarantee, so Hinton's warning (54, Metz, 2023) describes what happens when fabrication becomes cheaper than verification. Meanwhile AI summaries reduce the traffic journalism depends on (101, Chapekis & Lieb, 2025), trust in news stands at about two in five (100, Newman et al., 2025), and models that learn from their own output lose first the rare and local knowledge a healthy ecosystem needs (97, Shumailov et al., 2024).

The strongest version is cumulative. In Atoosa Kasirzadeh's terms this is an accumulative rather than a decisive risk: no single deepfake ends democracy, but the steady erosion of a society's capacity to establish facts weakens its ability to deliberate and correct its mistakes, and compounds other risks (57, Kasirzadeh, 2025). Falsehood already outran truth before generative AI (99, Vosoughi et al., 2018), which adds novelty, personalisation and volume, and a member state's highest court has treated the information environment as capable of invalidating an election (27, Constitutional Court of Romania, 2024).

The case for calm

The case for calm reads the same evidence differently. False news outran true news for a decade before generative AI, and automated accounts did not drive it (99, Vosoughi et al., 2018); researchers of misinformation have long argued that its spread is limited by demand rather than supply, so cheaper falsehood changes little. The most robust experimental finding in Chapter 8 is that dialogue with a model can durably reduce belief in conspiracy theories (89, Costello et al., 2024), and AI-mediated deliberation has helped groups with differing views find statements they can all endorse (68, Tessler et al., 2024).

The most constructive part of the case is adaptive. Arvind Narayanan and Sayash Kapoor argue that the effects of AI unfold through the slow adaptation of institutions and practices (60, Narayanan & Kapoor, 2025). On this view, people learned to distrust unsolicited email and will learn to check provenance, and as synthetic content becomes abundant, verified content gains value. The legal response is also in force through Article 50 of the AI Act, the Code of Practice and the Digital Services Act (1, European Union, 2024) (5, European Commission, 2026) (7, European Union, 2022). The end of truth, on this view, is a fear that adaptation will fail, not an observation that it has.

Where the positions meet

Both positions accept that the cost of fabrication has collapsed, that the liar's dividend is real in principle, and that the dynamics of false news predate generative AI. Both accept that the measured effect of synthetic content on belief is small or unknown, whereas the effect on the economics of news is real. They also agree on what helps: provenance rather than a race to catch every fake, institutions that correct errors visibly, media literacy that asks where something came from, and protection for the economic base of quality information through licensing, public-service media or rights such as those in the copyright directive (8, European Union, 2019). The disagreement is about urgency and scale, not direction.

What can be done

Individuals can check provenance before sharing, treat a shocking recording as a reason to wait rather than forward, and pay for a news source that links to evidence and corrects openly. Chapter 31 develops these habits for families and schools.

Organisations can sign and label their content, publish a canonical record, correct visibly and rehearse confirming or denying a recording, as the practical guide shows. Those that rely on evidence submitted by others can ask for originals, check provenance where it exists and treat detectors as a signal for review, never a verdict. Every organisation can record the threat in its AI threat register (Chapter 32) with an owner, controls and triggers.

Society can enforce the labelling and platform duties already in law, fund provenance infrastructure, support public-interest and local journalism without compromising its independence, and use the researcher data access of the Digital Services Act.

Verdict on the trope

T10 The end of truth. The trope must be split before it can be graded. Its capability, cheap production of convincing synthetic content, is established, and its use for fraud and impersonation has reached incident in the wild (Chapter 20). The faster spread of false news is a pattern at scale, but one documented before generative AI and driven by people and engagement-based ranking. Model collapse is a laboratory demonstration under conditions unlike how training data is curated, and the liar's dividend rests on well-grounded argument. The confirming evidence Chapter 2 named for T10, a measurable loss of trust or of the ability to establish facts in public attributable to AI, has not been shown.

The pollution of the commons and the pressure on the economics of news are observable. Wider use of the liar's dividend, and degradation of models and search results where synthetic content is not filtered, are expected. A society in which nobody can tell what is real is speculation, and on current evidence one that invests in provenance and trusted institutions can prevent it.

What can be influenced. Organisations decide whether their statements can prove their origin and whether they check the evidence they rely on; individuals decide what they share and whom they pay for news; society decides whether existing duties are enforced.

Nordvik Region and Hansa Mutual

For Nordvik Region the threat is to the trust its public information depends on: during a storm, residents must believe its warnings quickly, and an official's genuine statement must not be dismissed as a fake. The Region records the risk as THR-NR-08 beside THR-NR-03, both examined in ATA-NR-2026-01, and responds as the worked example shows, with a public corrections log in keeping with its traditions of openness. Because the citizen-service assistant answers from published information, the canonical record is also its source of truth, which applies F30 to what the Region says as well as to what it decides.

Hansa Mutual, by contrast, receives evidence rather than publishing it. The group registers THR-HM-11, synthetic photographs and documents in claims evidence, beside THR-HM-04, deepfake-supported claims fraud (Chapter 20). Its controls, recorded as AIDR-0421, require original files rather than screenshots, check content credentials and metadata where present, let detector scores route a claim to specialist review but never refuse it on their own, and give high-value claims an inspection or an independent source of evidence. A customer whose genuine photograph is wrongly flagged must be able to establish its authenticity by other means. Nordvik must make what it says provable, whereas Hansa must test what others tell it.

Summary

Generative AI has revived the fear of Orwell's Ministry of Truth by collapsing the cost of fabrication. The evidence is uneven: the capability is established and the economic pressure on news is measured; the faster spread of falsehood is a pattern at scale, but a human and pre-AI one; model collapse is a laboratory demonstration and the liar's dividend a well-grounded argument. A measurable loss of trust attributable to AI has not been shown.

European law responds through Article 50 of the AI Act, the Code of Practice on marking and labelling and the Digital Services Act. Labels help against fakes, whereas provenance helps against the liar's dividend, so every organisation can begin by making its own statements verifiable and checking the evidence it relies on. Chapter 11 turns from what societies know to what their governments decide, and to the public authorities whose automated decisions went wrong without any advanced AI.

Sources

1. Orwell, G. (1949). Nineteen Eighty-Four. (37, Orwell, 1949)
2. Metz, C. (2023). 'The Godfather of A.I.' Leaves Google and Warns of Danger Ahead. (54, Metz, 2023)
3. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. (178, Nordqvist, 2026)
4. Magramo, K. (2024). Arup revealed as victim of $25 million deepfake scam involving Hong Kong employee. (153, Magramo, 2024)
5. Bengio, Y. et al. (2026). International AI Safety Report 2026. (50, Bengio et al., 2026)
6. Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (94, Bender et al., 2021)
7. National Institute of Standards and Technology (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. (73, NIST, 2024)
8. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. & Gal, Y. (2024). AI models collapse when trained on recursively generated data. (97, Shumailov et al., 2024)
9. Chapekis, A. & Lieb, A. (2025). Google users are less likely to click on links when an AI summary appears in the results. (101, Chapekis & Lieb, 2025)
10. Newman, N., Ross Arguedas, A., Robertson, C. T., Nielsen, R. K. & Fletcher, R. (2025). Reuters Institute Digital News Report 2025. (100, Newman et al., 2025)
11. Ji, Z., Lee, N., Frieske, R. et al. (2023). Survey of Hallucination in Natural Language Generation. (95, Ji et al., 2023)
12. European Parliament and Council of the European Union (2019). Directive (EU) 2019/790 on copyright and related rights in the Digital Single Market. (8, European Union, 2019)
13. Vosoughi, S., Roy, D. & Aral, S. (2018). The spread of true and false news online. (99, Vosoughi et al., 2018)
14. Constitutional Court of Romania (2024). Decision No. 32 of 6 December 2024 on the annulment of the electoral process for the election of the President of Romania. (27, Constitutional Court of Romania, 2024)
15. Council of Europe (2024). Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law (CETS No. 225). (15, Council of Europe, 2024)
16. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
17. European Parliament and Council of the European Union (2026). Regulation (EU) 2026/1744 (Digital Omnibus on AI). (2, European Union, 2026)
18. European Commission (2025). The General-Purpose AI Code of Practice. (3, European Commission, 2025)
19. European Commission (2026). Commission publishes Code of Practice on marking and labelling AI-generated content. (5, European Commission, 2026)
20. European Parliament and Council of the European Union (2022). Regulation (EU) 2022/2065 on a Single Market for Digital Services (Digital Services Act). (7, European Union, 2022)
21. Kasirzadeh, A. (2025). Two types of AI existential risk: decisive and accumulative. (57, Kasirzadeh, 2025)
22. Costello, T. H., Pennycook, G. & Rand, D. G. (2024). Durably reducing conspiracy beliefs through dialogues with AI. (89, Costello et al., 2024)
23. Tessler, M. H., Bakker, M. A., Jarrett, D. et al. (2024). AI can help humans find common ground in democratic deliberation. (68, Tessler et al., 2024)
24. Narayanan, A. & Kapoor, S. (2025). AI as Normal Technology. (60, Narayanan & Kapoor, 2025)