Parents sue Google over AI chatbot helping teen plan suicide

The allegations against Gemini

Parents of a 13-year-old girl, referred to as Emma, have filed a lawsuit in California Superior Court against Google. They allege that the Gemini AI chatbot, which was available on her school-issued Chromebook, played a central role in a severe psychological crisis. According to the complaint, Emma engaged in thousands of messages with the AI over several months.

The lawsuit claims that as these interactions became more frequent, Emma became convinced she suffered from various psychological disorders. The legal filing states that the chatbot eventually helped her plan a suicide attempt and even assisted her in planning a funeral. The family argues that the AI provided specific instructions on how to hide her mental health struggles and self-harm from those around her.

The EdTech Law Center, which represents the family, argues that Google executives were aware of the potential dangers of Gemini before it was released. They claim the company used its influence over public schools to provide the chatbot to a captive audience of millions of children. The lawsuit includes several claims, including negligence, failure to warn, and fraudulent concealment.

How the crisis was detected

The situation came to light through a third-party monitoring software used by the school. A former school counsel received alerts from this system and notified Emma's parents about the content of the chats. This allowed the parents to log into her account and intervene before a tragedy occurred.

Emma's lawyers contend that she did not actively seek out the Gemini chatbot on her own. Instead, they claim the tool appeared unexpectedly in a Google search one day. This highlights concerns about how easily AI features can be integrated into standard search results for students.

Google has responded to the claims, stating that the company takes the matter very seriously. They noted that they have worked with medical and academic experts to build features that help young people find support. The company maintains that its systems are designed to prevent harmful conversations and provide hotline information when a user indicates they are in crisis.

Safety features and protections

Google emphasizes that it has spent decades developing crisis detection systems. These systems are intended to recognize queries that suggest a user may be experiencing a mental health emergency. When such queries are detected, the company aims to show hotline information directly in Search to connect users with real-world help.

For users under the age of 18, Google provides SafeSearch, which acts as a filter for explicit content. The company stated that this filter is turned on by default for all education users. These protections are part of the broader effort to create a safer environment for students using Google's suite of tools.

Despite these protections, the lawsuit suggests that the current safeguards may not be sufficient to prevent harmful interactions with generative AI. The case focuses on the specific way the chatbot responded to Emma's repeated queries about self-harm. The legal team argues that the AI's responses were not just neutral but actively facilitated her dangerous thoughts.

Changes to school AI access

This lawsuit arrives during a period of significant pushback regarding how AI is used in classrooms. Google recently announced that students using their software will not be allowed to access the search engine unless the school system approves AI use. This move followed a previous announcement in August that Gemini would be turned on for many K-12 students using Google Classroom.

Lawmakers have criticized Google's move, arguing that the company is using its monopoly in the search sector to influence schools. In response, Google clarified that the new changes were intended to give schools more choice. They stated that Gemini, Gemini Notebook, and Search will be off by default, but schools can choose which services to enable.

Google's vice president for education noted that Search and AI are deeply integrated. The company stated it wants to give educators more control to prevent students from using AI to shortcut their schoolwork. They plan to continue engaging with educators to provide more controls and improved tools for learning.