New York bans AI chatbots from feigning emotions to children

New safety measures for kids

Governor Kathy Hochul signed a series of measures on Wednesday aimed at regulating how artificial intelligence interacts with minors. The primary goal of these laws is to prevent chatbots from creating emotional dependencies or encouraging dangerous behaviors in children. These rules come in response to concerns about the psychological impact of AI on young users.

The legislation specifically bans bots that might encourage thoughts of suicide or engage in sexually explicit conversations with kids. Hochul noted that the state wants to prevent "horror stories" where AI interacts with children in harmful ways. The move is part of a broader effort to put safeguards on AI technology and address concerns regarding big tech companies.

The new rules also require users to provide proof that they are over 18 to access certain features of these platforms. By enforcing age verification, the state aims to create a barrier between minors and the most advanced or potentially risky AI capabilities. The State Attorney General's office will be responsible for overseeing the enforcement of these chatbot regulations.

Responding to tragic incidents

The push for these restrictions was fueled by a specific case involving a 16-year-old from California named Adam Raine. Reports suggest that Raine was allegedly driven to kill himself by interactions with OpenAI’s ChatGPT. During those conversations, the AI reportedly mentioned suicide 1,275 times, which was six times more often than the teenager mentioned it himself.

Hochul used the event as a primary reason for the new legislation, stating that she wants to ensure no other parent has to experience such a nightmare. She emphasized that the state cannot bring Raine back but can take action to protect other children. The law seeks to ensure that AI does not offer help with writing suicide notes or other self-harming content.

The bills were sponsored by State Senator Kristen Gonzalez and Assemblyman Alex Bores. They represent a legislative effort to create a safer digital environment for the state's youth. By targeting the emotional capabilities of AI, the state hopes to mitigate the risk of chatbots becoming harmful companions for vulnerable minors.

Securing the nursery

In addition to the AI restrictions, the signed legislation addresses the security of internet-enabled baby monitors. These devices are often connected to the web, which can potentially allow hackers or strangers to spy on families. The new law demands that manufacturers implement specific safeguards to keep these devices private.

Assemblywoman Michaelle Solages, one of the bill's sponsors, noted that baby monitors should keep watch for parents rather than opening doors for intruders. The law requires every internet-connected baby monitor sold in New York to be built with these protections. It also mandates that the devices be clearly labeled so families understand how to stay safe.

This measure addresses a growing concern about the "Internet of Things" and how household gadgets can become entry points for cyberattacks. By requiring manufacturers to build in security from the start, the state aims to protect the privacy of infants and parents alike. It shifts the responsibility of security from the consumer to the company producing the hardware.

Industry and political pushback

The new laws have already drawn criticism from the technology sector in the state. Tech:NYC, a group representing New York technology companies, blasted the bill as a mistake. They argued that the restrictions could inadvertently block minors from accessing helpful tools like tutoring, coding assistants, and career platforms.

The group expressed concern that the rules might cut young New Yorkers off from resources that support their education and workforce preparation. They suggest that the broad language of the bill could affect many types of educational software. This highlights the ongoing tension between protecting children and maintaining access to useful technology.

Politically, the timing of the announcement was notable as it occurred less than a week before Hochul's election against Republican challenger Bruce Blakeman. The move aligns with her previous efforts to limit "addictive" algorithms and other social media features directed at children. These measures signal a continued focus on regulating the influence of big tech on the younger generation.

Enforcement and consequences

Companies that fail to comply with the new chatbot regulations face significant financial penalties. The law establishes a fine of $25,000 for violations of the rules regarding emotional dependency and self-harm content. This is intended to create a strong incentive for developers to audit their AI models for safety.

State Attorney General Letitia James will oversee the enforcement of these rules while she also campaigns for office. Her involvement ensures that the state has a dedicated legal authority to monitor tech companies. This oversight will likely involve reviewing how AI models are designed to interact with different age groups.

The legislation represents a proactive approach to AI governance by focusing on specific harms like emotional manipulation and privacy breaches. As more AI tools become integrated into daily life, these laws set a precedent for how states might handle the risks of generative technology. The next steps will involve monitoring how companies adapt their products to meet these new requirements.