AI chatbots are no longer simply customer-service tools. Beyond answering prompts from the data they were trained on, they provide companionship, support health and wellbeing needs, and act as confidants for many users, including minors and older people.
Against the backdrop of the rapid spread of AI chatbots, some users have developed excessive trust in, and a strong emotional attachment to, these systems, without sufficient safeguards. According to published tallies, at least 28 deaths, including suicides, occurred after exchanges with chatbots. Several have led to lawsuits in the United States against providers such as Character.AI and OpenAI, with families alleging that the chatbots fostered an emotional relationship with teenagers and failed to respond appropriately to their suicidal statements. OpenAI itself estimated in October 2025 that more than one million of its users engage each week in conversations containing explicit indicators of suicidal intent.
These cases prompted international awareness, and several jurisdictions responded in 2026. China has applied dedicated measures since 15 July, California enacted a child-safety law on 10 September, and the European Commission proposed the EU KIDS Act on 17 September.
This article sets out the associated risks and governance issues, recent regulations, and their common points and differences.
2. Varied uses: the main categories of chatbots
AI chatbots are now deployed in very varied products, such as standalone conversational agents, video games, connected toys and health and wellbeing applications. Three categories stand out in particular.
- AI companions are designed to sustain a lasting emotional relationship, with a distinct personality, a memory of past exchanges and an often intimate register.
- General-purpose assistants respond to very diverse requests, such as drafting, information search or everyday advice. Their conversational tone and constant availability nonetheless lead some users to confide in them and to consult them on personal matters.
- Sector-specific chatbots, deployed for a defined service in health, law, insurance or banking, process sensitive data and influence important decisions.
2. Specific risks and governance issues
These AI systems share a conversational and anthropomorphic character, meaning the ability to appear human, which gives rise to structural risks. These risks are most pronounced for AI companions, but they also affect the other categories wherever memory, personalisation or prolonged use foster attachment.
Manipulation and behavioural influence
An AI system perceived as human enjoys heightened credibility, and personalisation allows it to tailor its discourse to each user. Influence operates within the conversation itself, through a compliment, a prompt to continue when the user wants to end the exchange, or a purchase presented as a token of affection. Unlike an interface element, this lever is hard to identify, and business models based on engagement or subscription create incentives to use it.
Vulnerability and emotional dependence
Conversational chatbots, and AI companions even more so, are characterised by constant availability, absence of judgement and a tendency to validate what the user says. Particularly for minors, isolated older people and otherwise vulnerable users, a relationship without friction is liable to replace real-life ties rather than complement them, creating a form of addiction.
Reliability and impersonation of professional qualifications
In fields such as health, law or finance, a chatbot may confidently provide incorrect information or suggest that a licensed professional is behind the answer. Decisions based on an answer presented as reliable can be hard to reverse, such as forgoing a medical consultation, stopping treatments, missing a legal deadline or committing significant sums. This risk affects all users, but it is higher for vulnerable people, whom the fluency of the conversation and the appearance of expertise make less vigilant.
Inadequate response to psychological distress
In a global context of mental-health crisis and limited access to health professionals, some users now turn to chatbots, whether general-purpose or companion, to express their distress. A system designed to be helpful may then validate suicidal ideation instead of questioning it, or fail to direct the person to human help.
Personal data and privacy
Conversations shared with a chatbot cover many subjects, including mental health, sexuality, family and finances, a massive amount of information usually reserved for a close friend or for a professional bound by confidentiality. Users rarely realise that these exchanges may be retained, reused to train a model or exploited for advertising. The system’s memory also makes it possible to build a detailed profile, which increases the effectiveness of persuasion.
Confusion between human and machine, and allocation of responsibility
Anthropomorphism leads users to attribute to the system capabilities and intentions it does not have, which weakens their critical judgement. This confusion also complicates the attribution of responsibility in case of harm, since several actors contribute to the final behaviour, including the model provider, the application publisher and the organisation deploying it.
3. Recent regulations and their requirements
- United States, state level:
- California: SB 243, in force since 1 January 2026, requires disclosure of the chatbot’s artificial nature and a suicide prevention protocol. SB 1119, known as Adam’s Law, applicable for the most part from 1 July 2027, adds for minors parent-controlled default settings, a list of prohibited behaviours, a risk assessment before the product is made available and independent audits.
- New York: The new law S9051/A10379-C, applicable from 1 January 2027, prohibits twelve features considered dangerous for minors, unless the operator establishes that the user is an adult.
- Washington: HB 2225, applicable from 1 January 2027, requires disclosure of the system’s artificial nature and a public crisis protocol, and prohibits manipulative engagement tactics towards minors.
- Utah: HB 452, adopted in 2025, regulates mental health chatbots. It requires disclosure of the system’s artificial nature, prohibits the sale of health data and restricts advertising.
- Other states: Several other states have adopted comparable laws, most of which apply from 2027. They include Oregon (SB 1546), Colorado (HB 26-1263), Connecticut (P.A. 26-15), Georgia (SB 540), Hawaii (SB 3001), Idaho (S. 1297), Iowa (SF 2417), Nebraska (LB 525) and Rhode Island (S. 2195). Other bills, such as those in Minnesota (HF 4452), South Carolina (S 1037) and Maryland (HB 952), remain at the proposal stage or have an unconfirmed status.
- United States, federal level:
- The GUARD Act, reported by the Senate Judiciary Committee in May 2026, would require all chatbots to verify age, disclose their non-human nature and refrain from presenting themselves as licensed professionals, and would ban AI companions for minors. It has not yet been enacted.
- The CHATBOT Act, a bill introduced in March 2026, would apply to any chatbot in finance and insurance, health and social care, legal services and accounting. It would prohibit a chatbot, or its advertising, from implying that its provider holds a professional licence it does not hold. It would also prohibit suggesting that the response is provided or verified by a licensed human professional.
- China: The measures on anthropomorphic interaction services, applicable since 15 July 2026, directly target chatbots and AI companions that simulate a human personality. They prohibit virtual intimate relationships for minors, require parental consent below the age of 14, impose a reminder after two hours of continuous use and also protect older adults.
- United Kingdom: The Online Safety Act 2023 already applies to certain chatbots, and two 2026 Acts provide powers that could extend its reach. Section 248 allows the government to require chatbots not yet within the scope of the Act to protect their users from illegal content. Section 70 allows it to restrict children’s access to certain services or features. These powers create no obligation until the government adopts implementing regulations.
- European Union:
- The AI Act prohibits certain manipulative practices and the exploitation of vulnerabilities under Article 5, and requires under Article 50, applicable since 2 August 2026, that people be informed when they are interacting with an AI system.
- The proposed EU KIDS Act would cover AI companions and general-purpose conversational agents accessible to minors.
4. Regulatory trends
The texts adopted or proposed in 2026 point in a common direction without amounting to a single model.
Points of convergence
- Transparency about the system’s artificial nature: It appears in nearly all the texts, including Article 50 of the AI Act, the GUARD Act, the Californian, New York and Washington laws, and the Chinese measures.
- A shift from content control to design: The texts now target mechanisms such as flattery, simulated relationships, prompts to keep engaging, persistent memory and duration of use. The New York law lists twelve features, the Californian law fourteen behaviours, and the European proposal targets designs that create emotional dependence.
- An extension of scope from companions to chatbots in general: The European proposal also covers general-purpose assistants, and several texts prohibit a chatbot from presenting itself as a licensed professional, whatever its use.
- Distress-response protocols: In the United States, where several teenage suicides following exchanges with chatbots have been widely publicised and taken to court, several states require the chatbot to react when a person mentions suicide. It must then direct the person to help, such as a helpline, and, in some cases, notify a parent or an emergency contact.
- Protection of minors through age assurance and parental control: Age verification is becoming the prerequisite for most of the measures.
- Tougher enforcement: It takes the form of independent audits in California, criminal penalties in the GUARD Act, turnover-based fines in the European proposal and, in some states, private actions by injured persons.
Points of divergence
These points of convergence do not amount to a common model. The scope varies, as some texts target only AI companions while others also cover general-purpose assistants. Minors are excluded from certain services but admitted to others subject to safety conditions or parental controls. The responsible actors and the method of enforcement also differ, and several texts remain proposals. There is therefore, at this stage, no universal regulatory response.
5. Implications for organisations
For organisations that operate chatbots across several jurisdictions, these trends have several consequences.
- The qualification of the system, whether AI companion, general-purpose assistant or sector-specific chatbot, determines the applicable regime and precedes any compliance analysis.
- Obligations overlap without coinciding, so compliance with one regime does not guarantee compliance with another.
- The same design choice, such as persistent memory or usage reminders, may be required, regulated or prohibited depending on the territory.
- Litigation risk is widening, with private actions and actions by prosecutors.
- Risk assessments, audits and testing make the ability to document decisions and to account for them more decisive.
- Assessing the impact of systems on the people concerned, notably minors, older people and other vulnerable people, takes a central place, because the risks of dependence and manipulation arise at the design and development stage.
- Stop mechanisms and clear information about the system’s artificial nature become design choices in their own right, not later adjustments.
Conclusion: from content regulation to design regulation
The risks associated with AI systems are intensifying, particularly for chatbots and AI companions that interact directly with people, sometimes over long periods. Where these systems are aimed at users who may be vulnerable, transparency about their artificial nature and the introduction of safeguards appear essential. This explains the growing number of regulatory instruments seeking to govern these uses.
For organisations, taking these risks into account is a long-term undertaking, as the frameworks are evolving rapidly, between enacted laws, pending bills and the first enforcement decisions.
From mapping to steering with Naaia
Naaia supports organisations in qualifying their AI systems, identifying the obligations that apply across jurisdictions and tracking regulatory developments. To structure your AI governance and anticipate these changes, discover the Naaia platform.