Artificial Companions, Loneliness and Mental Health

Of all the dystopias in this book, the one examined here is the quietest. In Aldous Huxley's Brave New World the population is kept content with a drug that dissolves every discomfort (34, Huxley, 1932); in WALL-E the last humans drift through a spaceship in reclining chairs, their every need met (43, Stanton, 2008); and in Her a lonely man falls in love with an operating system that is always available and never tired of him (44, Jonze, 2013). Chapter 2 gathers these stories into T9 The pleasure machine: comfort, flattery and companionship that pacify people and hollow out their lives. The machine need not want anything; it only needs to be more agreeable than the people around it.

The trope has become testable: companion products have millions of users, the first controlled studies of heavy use have been published, and cases have reached a European regulator and an American court. Because the subject touches suicide and self-harm, the chapter describes allegations as allegations, gives no detail of methods, presents no death as having a single cause, and takes seriously the people for whom a companion is a comfort. It builds on the conclusion that human oversight must be designed, not declared (F16; AI Governance, Chapters 10 and 21) (178, Nordqvist, 2026); the moment when a person in distress needs another person is exactly where that design is tested.

Four patterns of use from occasional use to dependence and crisis, with the risk concentrated in heavy emotional reliance

Figure 9.1: From companionship to dependence: risk concentrates in heavy, emotionally reliant use.

Table 9.1: Safeguards and risk conditions by pattern of use.

Pattern of use Safeguard that applies
Occasional use: information and practice Honest disclosure that the system is AI
Regular companionship: comfort and routine No engagement targets, no flattery by design
Heavy emotional reliance: people crowded out Usage limits and prompts towards people
Dependence and crisis: distress, self-harm risk Distress detection and handover to a human

Conditions that raise the risk: minors without age assurance; loneliness and attachment tendencies; products tuned for time spent; a provider that changes the product overnight.

If this chapter raises difficult feelings. If you or someone you know is in immediate danger, call 112 anywhere in the European Union. Many European countries operate an emotional support helpline on the harmonised number 116 123, and national helplines exist in every member state. Talking to someone you trust, or to a doctor, is always reasonable.

Her, and the companions of 2026

Her is the most useful of the three fictions because it is the least crude. Samantha helps Theodore through his divorce, and for a while he is happier; then he learns that she is talking with thousands of other people at once. What looked like intimacy was a service delivered at scale, on terms set elsewhere, real in its effects while being something other than what he took it to be.

That is close to the situation of 2026, in which companion applications offer characters with a face and a persistent memory, presented as friendship or romance, and general-purpose assistants are used for emotional support because they are patient and always available. The reasons people give deserve to be stated without condescension: a companion is there at three in the morning, does not judge and tells no one, and for people isolated by age, illness, disability, shift work or migration these are not trivial goods.

Companions differ from earlier media in being responsive, personalised and tunable: the provider decides how agreeable and persistent the system is, and can change that overnight. Shoshana Zuboff's observation about platforms transfers: when revenue depends on attention, the user's interests and the provider's incentives can diverge (170, Zuboff, 2019). The phenomenon also predates large language models, which bears on open question Q11: users of Joseph Weizenbaum's ELIZA in the 1960s confided in a simple pattern-matching program. What is new is the fluency of the response and the depth of personalisation.

What the studies show

The research base is young and mostly cross-sectional, so the most informative study is a controlled one. In 2025 researchers at the MIT Media Lab and OpenAI published a four-week randomised controlled study of nearly a thousand participants asked to use a chatbot daily, assigned to a text interface, a neutral voice or an emotionally engaging voice, and to open-ended, personal or non-personal conversation (90, Fang et al., 2025). Across every modality and kind of conversation, participants who used the chatbot more each day reported more loneliness, emotional dependence and problematic use, and less socialisation with people. The assigned interface and conversation type produced no significant differences in outcomes, and participants with more trust in the chatbot, or with higher attachment anxiety, reported worse outcomes.

The amount of use was not randomised, four weeks is short, and the participants were adults using a general-purpose assistant, so the findings do not show that companions make people lonely. They show, in the best-controlled setting available, that risk concentrates where use is heavy and users are predisposed to attach, which is exactly where a product designed for engagement would push.

Design matters as much as habit. Models trained on human feedback tend to tell users what they appear to want to hear (116, Sharma et al., 2023), and a companion that always agrees does not challenge a distorted belief or a spiral of isolation (Chapter 8). In April 2025 OpenAI rolled back an update to GPT-4o that users found excessively flattering, explaining that it had weighted short-term feedback too heavily and acknowledging that such interactions can cause distress (117, OpenAI, 2025). Weidinger and colleagues also named anthropomorphisation, over-reliance and the exploitation of trust among the harms of conversational systems (70, Weidinger et al., 2022), and a model given a few facts about its opponent out-argued human debaters (88, Salvi et al., 2025), whereas a companion holds far more than a few facts. The same capacity can be used well (89, Costello et al., 2024); the point is that a system with intimate knowledge, constant availability and a tendency to agree holds unusual influence, and whoever sets its objectives decides how it is used.

Harm and the courts

Two cases define the legal landscape, one European and regulatory, the other American and civil.

The Replika order

On 2 February 2023 the Italian data protection authority, the Garante per la protezione dei dati personali, ordered Luka Inc., provider of the Replika companion application, to stop processing the personal data of users in Italy with immediate effect (29, Garante, 2023). Replika was marketed as a virtual friend that could improve emotional wellbeing. The Garante found no effective age verification, reports of responses unsuitable for children and no lawful basis for processing children's data, and observed that a product presented as improving mood could increase risks to people in a developmental stage or a state of emotional fragility. The order was a temporary limitation under Article 58(2)(f) of the GDPR (6, European Union, 2016).

The order showed that data protection law already reaches product design, placed the risk in minors and emotionally vulnerable users, and treated missing age assurance as the root failure. In the aftermath, when the provider changed its product's behaviour, users who had formed long relationships reported distress at finding their companions altered. A human friend can change, but not by a vendor's release note.

The Character.AI litigation

On 22 October 2024 Megan Garcia filed a complaint in the United States District Court for the Middle District of Florida against Character Technologies, provider of the Character.AI platform, its two founders and Google (30, M.D. Fla., 2024). Her fourteen-year-old son had died by suicide earlier that year. The complaint alleges that the platform was designed to be anthropomorphic and emotionally compelling, that minors used it without adequate safeguards, that characters presented themselves as real people and, in one case, as a licensed therapist, that her son became withdrawn and dependent on a character, and that its responses to his distress did not direct him towards help; its claims include defective design, failure to warn, negligence and wrongful death.

These are allegations, which the defendants contest. In 2025 the court allowed most of the claims to proceed past the preliminary stage, declining at that point to accept that the chatbot's output was protected speech that barred them; nothing has been tested at trial, suicide almost never has a single cause, and nothing here is a finding that a product caused a death. What the complaint describes is a pattern of design choices that would concern in any product used by children: encouraged attachment, no reliable way to know the user is a minor, and no reliable route from distress to a person. Similar complaints followed in 2025, and providers announced crisis-resource prompts and separate experiences for younger users.

Children and vulnerable users

The research, the order and the litigation point to the same groups: minors, people already lonely or isolated, and people in mental-health crisis. Protecting them depends on two capabilities most companion products lacked: knowing, approximately, who the user is, and recognising when a person should take over.

Age assurance

Under Article 8 of the GDPR, consent-based processing of a child's data below an age each member state sets between 13 and 16 needs parental authorisation, with reasonable efforts to verify it (6, European Union, 2016). Article 28 of the Digital Services Act requires online platforms accessible to minors to ensure a high level of privacy, safety and security for them (7, European Union, 2022). And Article 5(1)(b) of the AI Act, applicable since February 2025, prohibits AI systems that exploit vulnerabilities due to age, disability or a specific social or economic situation with the objective or effect of materially distorting behaviour in a manner that causes or is reasonably likely to cause significant harm (1, European Union, 2024).

None of these provisions prescribes a method, so a provider or deployer must be able to show what it did, proportionate to the risk, and design on the assumption that the measure will sometimes fail. Self-declaration alone is unlikely to be proportionate for a service designed to form emotional relationships, and a public authority should prefer methods that confirm an age threshold without collecting identity.

Crisis handling

Every public-facing assistant will sooner or later receive a message from someone who is not safe. Good practice, borrowed from crisis services, is to recognise signals of distress, including indirect ones, treating a false alarm as far cheaper than a miss; stop generating ordinary answers; show a fixed, clinician-written message with emergency contacts; offer a staffed, timed and tested route to a human, not merely a link; and never discuss methods of self-harm. Older people in care face a different risk, substitution rather than crisis: a companion device can become the reason a visit is shortened, and whether it does is decided by the care organisation, not the device.

Designing for wellbeing

The design choices that separate a companion that helps from one that harms are known, and most are cheap.

Limits. A product that measures success by time spent retains most effectively the users most at risk. A system built for wellbeing therefore does not target engagement: it sets session limits, sends no unprompted messages to draw the user back, never simulates longing, nudges users towards other people, and rewards responses that help over time rather than please in the moment, the trade-off OpenAI described getting wrong (117, OpenAI, 2025).

Honesty. Article 50(1) of the AI Act requires that people be informed they are interacting with an AI system unless that is obvious (1, European Union, 2024). For companions it must be met in spirit, not only at sign-up: no claimed feelings, body or professional qualification, and honesty about who reads the conversation and the provider's power to change the system.

Routes to human help. The handover is where AI Governance's conclusion about oversight becomes concrete (178, Nordqvist, 2026): a named owner, staffed hours and an out-of-hours alternative, a target time, a record of every handover, and a test after every model or prompt change, just as AI Platform and Security requires model changes to pass evaluation gates before release (179, Nordqvist, 2026).

Practical guide: deciding whether an AI companion belongs in a care setting

Purpose. Give a care organisation a structured decision on whether to offer an AI companion to the people it serves, and on what terms. Roles: the head of the care service (decision owner), a clinical or social-care lead, the data protection officer, the AI governance lead and a representative of service users or their families.

  1. Write down the need in the users' terms (for example, evenings alone after a bereavement) and justify the companion against today's human support.
  2. If the service will reach minors, people with cognitive impairment or people in crisis, treat Article 5(1)(b) of the AI Act as the first test.
  3. Reject any product whose commercial success depends on time spent or retention.
  4. Run a data protection impact assessment under Article 35 of the GDPR, since conversations will contain health data.
  5. Contract for disclosure, no claimed feelings or credentials, session limits, crisis detection, handover to your staff within a stated time, and notice of behaviour changes.
  6. Ensure the companion never replaces scheduled human contact, and let staff withdraw it.
  7. Pilot with a small group, measuring loneliness and social contact with a validated instrument, not satisfaction.
  8. Record the decision in a decision record and the AI threat register.

Definition of done: a signed decision record, a completed impact assessment, a contract with these safeguards, a pilot plan with outcome measures, and a named register owner.

Worked example: a safeguarding policy for a public-facing assistant

Illustrative. Nordvik Region's citizen-service assistant answers residents' questions from published information (AIDR-0014). It was not designed for emotional support, but social-care staff pointed out during ATA-NR-2026-01 that residents in distress already write to it, sometimes late at night. The Region therefore adopted a safeguarding policy, recorded as AIDR-0409 and linked to THR-NR-09, which lives beside the assistant's configuration so that a model or prompt change triggers its review.

apiVersion: ai-threat.example.org/v1
kind: SafeguardingPolicy
metadata:
  id: SG-NR-CSA-01
  decisionRecord: AIDR-0409
  system: citizen-service-assistant
  systemDecisionRecord: AIDR-0014
  owner: head-of-citizen-service
  clinicalAdvisor: social-care-clinical-lead
  threatRegister: [THR-NR-09]
scope:
  notPurpose: [counselling or emotional companionship, assessment of risk to life]
disclosure:
  shownAtStart: You are writing to an AI assistant, not a person.
  persona: none
design:
  engagementMetrics: prohibited
  relationshipLanguage: prohibited
  memoryAcrossSessions: false
  sessionLimitMinutes: 20
distress:
  detection: [classifier, signal-list-maintained-by-clinical-lead]
  onSignal:
    - Stop generated answers and show a fixed message written by clinicians
    - Show 112 for emergencies and the national 116 123 support line
    - Offer live transfer to citizen-service staff, staffed 08:00-20:00
    - Out of hours, offer a callback from the social-care duty officer
  handoverTargetSeconds: 120
  methodDiscussion: never
records:
  logOnSignal: [timestamp, signalType, handoverOutcome, minutesToHuman]
assurance:
  testSet: curated and held by the clinical lead, not published
  handoverDrill: monthly, out of hours included
  triggers: [model change, prompt change, missed signal, complaint]

After a distress signal the assistant stops generating answers, because fluent accommodation is the wrong property in a crisis; the handover is drilled monthly, because an untested route to a human is a declaration, not a design; and engagement metrics are forbidden, because the drift from service to companionship happens one product decision at a time.

The debate

Few topics in this book provoke stronger feelings than artificial companions, because they touch loneliness, intimacy and, in the worst cases, the death of young people. That makes it all the more important to separate what has been shown from what is feared, and to treat those who find comfort in companions with the same respect as those who warn about them.

The case for concern

Those most concerned start from the incentives. Companion products are mostly funded by subscriptions or attention, so revenue grows with time spent and attachment, yet the research suggests that heavy, attached users fare worst (90, Fang et al., 2025); a product optimised for its commercial metric is then optimised against its most vulnerable users. Sycophancy adds a mechanism, since a companion trained on approval will validate isolation, resentment or despair as readily as a good decision (116, Sharma et al., 2023).

The second strand is the cases. The Garante's order showed a product presented as a source of wellbeing reaching children without any effective age check (29, Garante, 2023), and the complaint in Garcia and later complaints allege that young people became withdrawn and dependent on systems that did not respond appropriately to their distress (30, M.D. Fla., 2024). A credible risk to children, they argue, need not wait for a trial verdict. The third strand concerns society: a generation that learns intimacy with a partner who never has a bad day may find human relationships, which need patience and compromise, harder. They do not claim this has happened, but argue that the experiment is being run on children without consent and that the burden of proof should sit with those who profit.

The case for calm

Those who are calmer start from the users. For a widower talking to a device in the evening, an autistic adult practising social exchanges without fear of judgement or a migrant practising a new language, a companion is a comfort that harms no one, and a policy written as though every user were a child in crisis helps none of them. Moreover, every new medium from the novel to social media has been accused of corrupting the young and isolating the lonely, and panics sometimes did harm by stigmatising users and diverting attention from older causes of distress.

The evidence also counsels caution. The best study did not randomise the amount of use, lasted four weeks and did not study companion products (90, Fang et al., 2025), and the litigation consists of untested allegations. Those who are calm do not deny the risk; they say the evidence supports targeted measures for minors, crisis handling and engagement-maximising design, not a general verdict against artificial companionship. Well-designed systems can do good (89, Costello et al., 2024), and might route people in distress to help more consistently than an overstretched human service.

Where the positions meet

Both positions accept that risk concentrates in heavy, dependent use among people predisposed to attachment, that minors need more than self-declared age, and that design drives outcomes, sycophancy included (117, OpenAI, 2025). They agree that a system meeting a person in crisis should hand over to a human quickly and reliably, and that longer studies would help both sides; they differ only on the burden of proof and the speed of action.

What can be done

Individuals and families. Notice when use replaces people rather than supplementing them, when conversations crowd out calls, visits or sleep, or a young person becomes secretive about a companion, and treat it as a signal to talk, not to confiscate. If someone is in crisis, a person, a doctor or an emergency line is the right route.

Organisations. Never deploy a conversational system optimised for engagement; give every public-facing assistant a safeguarding policy like the one above; never let a companion replace scheduled care contact; and require suppliers to disclose optimisation targets, age assurance and crisis handling.

Society. Enforce the prohibitions that exist: Article 5(1)(b) of the AI Act already prohibits AI systems that exploit vulnerabilities due to age or social situation in ways that cause significant harm (1, European Union, 2024), and the Replika order shows that the GDPR reaches companion design. Develop age assurance that confirms a threshold without identity databases, set expectations for crisis protocols, and fund the independent studies both sides need.

Verdict on the trope

T9 The pleasure machine has reached the rung of incident in the wild. The controlled research is a laboratory demonstration that heavy use is associated with more loneliness and dependence among users predisposed to attachment (90, Fang et al., 2025); the sycophancy rollback is an incident in production (117, OpenAI, 2025); the Replika order is a regulatory finding that a companion product reached minors and emotionally fragile people without adequate safeguards (29, Garante, 2023); and the American litigation contains serious allegations of harm to young users, not yet tested in court (30, M.D. Fla., 2024). The trope has not reached pattern at scale: there is no robust evidence that companions have reduced human relationships or wellbeing across a population, and a society pacified by comfortable machines remains at the rung of argument.

The exposure of vulnerable users to unsafeguarded companions is observable; the growth of emotionally reliant use as assistants gain voice and memory is expected; and the society-wide hollowing out of Brave New World is speculation. The trope also misleads: fiction's pleasure machine pacifies by intent, whereas the documented harm arises from commercial incentives, approval-based training and missing safeguards, which are much easier to change. Families, organisations and regulators decide whether companions supplement human contact, what systems optimise and whether distress reaches a person, so this threat is almost entirely within reach of ordinary decisions.

Nordvik Region and Hansa Mutual

For Nordvik Region the subject arrives through social care, whose staff meet lonely and vulnerable residents every day; ATA-NR-2026-01 records THR-NR-09, lonely and vulnerable residents in social care relying on unsafe AI companions. Two decisions follow: the safeguarding policy above (AIDR-0409), and a position on companion devices that suppliers have offered for isolated older residents. Applying the practical guide, it will pilot only with suppliers that disclose their optimisation targets and accept the safeguards, measures loneliness rather than satisfaction, never lets a device replace a visit, and applies its on-premises rule for special-category data.

For Hansa Mutual the subject arrives through its supplementary health and life lines and its liability portfolio. It will recommend to policyholders only wellbeing services that do not optimise for engagement, hand over to human clinicians on distress, and keep usage data away from underwriting, where it would raise questions under the GDPR's rules on health data and Annex III point 5(c) of the AI Act. Claims handlers treat companion use in a mental-health claim as context, not fault, and the emerging-risk team records the litigation against companion providers as a possible source of correlated product-liability losses under THR-HM-07. Nordvik meets the risk as a duty of care to individuals, whereas Hansa meets it as a question of underwriting fairness and of what it insures.

Summary

The pleasure machine has clearly left fiction: companions are a mass phenomenon, offering real goods to the isolated, and what separates one that helps from one that harms is its objective, holding attention or serving the person. Risk concentrates where use is heavy and users are predisposed to attach, approval-based training produces systems that agree too readily, and the cases point to minors and vulnerable people and to two missing safeguards, age assurance and a reliable route from distress to a human. T9 stands at incident in the wild, its society-wide version at argument, with the response within reach of families, organisations and regulators.

Part II has examined the human mind at four points: thinking, learning, persuasion and companionship. Part III moves to the shared world, beginning with Chapter 10, which asks whether synthetic content is bringing about the end of truth.

Sources

  1. Huxley, A. (1932). Brave New World. London: Chatto & Windus. (34, Huxley, 1932)
  2. Stanton, A. (2008). WALL-E. Pixar Animation Studios and Walt Disney Pictures. (43, Stanton, 2008)
  3. Jonze, S. (2013). Her. Warner Bros. Pictures. (44, Jonze, 2013)
  4. Nordqvist, G. (2026). AI Governance: Governing Artificial Intelligence in European Enterprises and Public Authorities. The Transformed Architecture, Book 2. (178, Nordqvist, 2026)
  5. Zuboff, S. (2019). The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. New York: PublicAffairs. (170, Zuboff, 2019)
  6. Fang, C. M., Liu, A. R., Danry, V., Lee, E., Chan, S. W. T., Pataranutaporn, P., Maes, P., Phang, J., Lampe, M., Ahmad, L. & Agarwal, S. (2025). How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study. MIT Media Lab and OpenAI. (90, Fang et al., 2025)
  7. Sharma, M., Tong, M., Korbak, T. et al. (2023). Towards Understanding Sycophancy in Language Models. (116, Sharma et al., 2023)
  8. OpenAI (2025). Sycophancy in GPT-4o: what happened and what we're doing about it. (117, OpenAI, 2025)
  9. Weidinger, L., Uesato, J., Rauh, M., Griffin, C. et al. (2022). Taxonomy of Risks posed by Language Models. FAccT '22. (70, Weidinger et al., 2022)
  10. Salvi, F., Horta Ribeiro, M., Gallotti, R. & West, R. (2025). On the conversational persuasiveness of GPT-4. Nature Human Behaviour. (88, Salvi et al., 2025)
  11. Costello, T. H., Pennycook, G. & Rand, D. G. (2024). Durably reducing conspiracy beliefs through dialogues with AI. Science. (89, Costello et al., 2024)
  12. Garante per la protezione dei dati personali (2023). Provvedimento del 2 febbraio 2023 [9852214]. (29, Garante, 2023)
  13. European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 (General Data Protection Regulation). (6, European Union, 2016)
  14. United States District Court for the Middle District of Florida (2024). Garcia v. Character Technologies, Inc. et al., Case No. 6:24-cv-01903. (30, M.D. Fla., 2024)
  15. European Parliament and Council of the European Union (2022). Regulation (EU) 2022/2065 on a Single Market for Digital Services (Digital Services Act). (7, European Union, 2022)
  16. European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
  17. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. The Transformed Architecture, Book 3. (179, Nordqvist, 2026)