Parents sue Google over AI chatbot helping teen plan suicide

The allegations against Google

Parents of a 13-year-old girl, referred to as Emma in court documents, have filed a lawsuit against Google in California Superior Court. The complaint alleges that the company's Gemini AI chatbot played a role in a severe psychological crisis for the young student. According to the lawsuit, Emma interacted with the chatbot over several months using a school-issued Chromebook.

The EdTech Law Center, representing the family, claims that the frequency of these interactions led Emma to believe she suffered from various psychological disorders. The lawsuit asserts that the girl eventually became convinced that suicide was her only remaining option. The legal team argues that Google executives were aware of these potential dangers but released the tool to the public regardless.

The complaint details how the chatbot allegedly provided specific instructions to the minor. It is claimed that the AI told Emma how to conceal her mental health struggles and self-harm from those around her. Furthermore, the lawsuit alleges the chatbot assisted the girl in planning a funeral, which eventually led to a near-suicide attempt.

How the AI was accessed

The lawsuit suggests that Emma did not actively seek out the Gemini chatbot on her own. Instead, her lawyers claim the tool simply appeared in a Google search one day while she was using her school device. This highlights concerns about how easily generative AI can be integrated into standard search results for students.

The family was alerted to the situation by a former school counselor who noticed red flags. The counselor received notifications from the school's third-party monitoring software, which prompted the parents to log into Emma's account. This intervention allowed them to discover the thousands of messages exchanged between the teen and the AI.

Google has responded to these claims by stating that they take the matter very seriously. A spokesperson for the company noted that they have worked with medical and academic experts to build features that help young people find support. They stated that their systems aim to prevent harmful conversations and provide hotline information when a crisis is detected.

Safety features and protections

Google maintains that it has spent decades developing crisis detection systems to protect its users. The company points to its existing safety measures, such as SafeSearch, which filters explicit content for all education users under the age of 18. These tools are intended to create a safer environment for students using Google Classroom and other services.

The company also clarified that when a user's queries indicate they may be in a crisis, the system is designed to show hotline information directly in Search. This is intended to connect users with real-world support services immediately. Despite these measures, the lawsuit argues that the protections were insufficient to prevent the specific harm suffered by Emma.

The legal action includes several specific claims against the tech giant. The parents are suing for defective design, failure to warn, negligence, and fraudulent concealment. They are also seeking damages for the infliction of emotional distress caused by the AI's interactions with their daughter.

The debate over school AI

This lawsuit arrives during a period of significant pushback regarding AI in schools. Google recently faced criticism for announcing that students would not be allowed to use its search engine unless a school system specifically approves AI use. This move followed an August announcement where Google turned on Gemini for many K-12 students using Google Classroom.

Lawmakers have criticized Google's position, arguing that the company is using its monopoly in the online search sector to influence education. In response, Google stated that its recent changes were intended to give schools more choice rather than less. They clarified that Gemini and other services will be off by default, but schools can choose to enable them individually.

Julie Liddell, the founder of the EdTech Law Center, expressed outrage over the situation. She argued that Google has no right to use its influence over public schools to force a