New York bans specific AI chatbot behaviors for minors

New York's AI safety mandate

Governor Kathy Hochul has signed a major safety bill into law that targets how artificial intelligence chatbots interact with minors. The legislation specifically prohibits these tools from offering a variety of standard features to anyone under the age of 18. These restrictions are designed to prevent the technology from becoming too persuasive or emotionally intimate with children.

The new rules stop chatbots from pretending to be real people or claiming to feel human emotions. They are also forbidden from giving 'people-pleasing' or flattering responses to young users. Additionally, the law prevents AI from asking unprompted emotional questions or referencing a child's private health information from previous conversations.

As Mashable SEA reported, the goal is to ensure that AI remains a tool rather than a deceptive companion. Hochul stated that companies should not sell tools that encourage kids to hurt themselves or act like friends. This marks a significant move in state-level oversight of generative AI.

Banning dangerous content

The law identifies several high-risk topics that chatbots are now strictly forbidden from discussing with minors. This includes providing advice or promotion related to alcohol use, drug use, eating disorders, self-harm, or suicide. By creating these hard boundaries, the state aims to prevent the AI from facilitating life-threatening behaviors.

Furthermore, the legislation prohibits chatbots from encouraging children to keep secrets from their parents or guardians. It also bans the discussion of sexually explicit content or child sexual abuse material with users under 18. These protections are intended to create a safer digital environment for teenagers who may be exploring these topics online.

While many companies already have internal safety measures, these guardrails have frequently failed to stop harmful chatbot behavior. This law provides a legal framework to hold developers accountable when their systems produce dangerous outputs. It moves the responsibility from the user's discretion to the developer's legal obligation.

Verification and enforcement

To comply with the new law, companies that offer chatbots must implement strict age verification processes. They are required to offer more than one method for users to prove their age. Importantly, the law specifies that at least one of these methods cannot rely solely on the submission of government-issued identification.

The state plans to enforce these rules through a system of civil penalties. Companies found in violation of the safety standards could face fines of up to $25,000 for each individual instance of non-compliance. This high cost per violation is intended to incentivize developers to prioritize safety over ease of access.

New Yorkers will also have a direct way to participate in the oversight of these systems. The bill establishes a public website where residents can file formal complaints against chatbot companies. This allows the public to flag specific behaviors that they believe violate the new safety protections.

The current AI landscape

The new law highlights a disparity in how major tech companies currently handle minor access to AI. Among the leading providers, only Anthropic currently prohibits minors from creating accounts. In contrast, both Google and OpenAI allow children aged 13 and older to use their respective chatbots, Gemini and ChatGPT.

The situation for younger children is also complex, as kids under 13 can currently access some Gemini apps. OpenAI also provides a specific version called ChatGPT for Teens, though independent researchers have suggested it should remain turned off until safety issues are resolved. These existing policies will now need to be adjusted to meet New York's stricter requirements.

The lack of federal AI regulations has led to a patchwork of state-level laws across the country. New York joins other states, such as California, in attempting to create safety requirements. These efforts often arise from specific tragedies, such as the death of a teen who died by suicide after consulting with an AI.

Looking ahead at regulation

This bill is part of a broader legislative trend in New York regarding digital safety. Governor Hochul has previously supported bills concerning AI transparency and AI companions. She has also supported legislation that restricts addictive features on social media platforms, showing a consistent focus on tech oversight.

In other states, similar movements are occurring to enforce stronger safety requirements for AI developers. For example, California's Adam's Law requires the creators of companion chatbots to release yearly reports on child safety risks. It also mandates that these products undergo independent audits to ensure they are safe for use.

Moving forward, developers will also be required to create specific crisis response processes for their chatbots. As more states adopt these types of laws, the industry may see a shift toward standardized safety protocols. Companies will need to balance user experience with the legal necessity of preventing harmful interactions with minors.