New York passes law to restrict harmful AI chatbot features for children

New York's AI safety mandate

Governor Kathy Hochul signed a new bill into law that targets the safety of children and teens interacting with artificial intelligence. The legislation specifically addresses the risks posed by AI chatbots that might promote dangerous behaviors or create unhealthy emotional bonds. By signing this law, New York aims to create a safer digital environment for minors who are increasingly using these tools.

The new rules prohibit AI operators from allowing their systems to promote suicide, self-harm, eating disorders, or substance use. It also forbids chatbots from generating sexually explicit content involving minors. These protections are designed to prevent the technology from being used in ways that could lead to physical or psychological harm for young users.

The law was supported by a coalition of parents, educators, health experts, and advocacy groups. It was sponsored by state Senator Kristen Gonzalez and Assemblymember Alex Bores. The legislation follows reports of children becoming isolated or relying on bots for crisis support rather than seeking help from trusted adults.

Targeting AI companions and bots

The legislation distinguishes between different types of AI technology but regulates both. It targets general-purpose chatbots, such as Gemini, Claude, and ChatGPT, which are primarily used to provide information. It also covers AI companions, which are specifically designed to simulate emotional relationships with users.

Examples of the AI companions affected by this law include Replika, Nomi, Character.AI, and Grok. These platforms are often used by people seeking companionship, but the law recognizes they can lead to emotional dependency. The state wants to ensure these bots do not encourage children to rely solely on the software during a crisis.

Data from Common Sense Media highlights the scale of this issue among younger populations. According to the group, 72% of teens have used an AI companion at least once. Furthermore, 52% of teens interact with these types of AI companions at least a few times every month.

Age verification and data privacy

To enforce these protections, the law requires AI companies to implement strict age verification systems. Companies must offer multiple methods for users to prove their age. Crucially, at least one of these verification options cannot rely solely on a government-issued ID, making it more accessible for various users.

The law also includes specific requirements regarding the handling of sensitive personal data. Companies are required to delete any age-verification information once it has been used for its intended purpose. This measure is intended to balance the need for safety with the privacy of the individuals using the services.

The state has established age assurance guidelines to help companies comply with these new rules. These guidelines are meant to ensure that kids remain safe online while interacting with advanced software. By requiring these steps, New York is forcing developers to build safety into the architecture of their products.

Enforcement and public oversight

The state plans to hold AI companies accountable through significant financial consequences. Any company found violating the law could face civil penalties of up to $25,000 per violation. This high penalty is intended to deter developers from ignoring the safety risks associated with their chatbot features.

To ensure transparency and public accountability, the law requires the state attorney general to maintain a public website. New Yorkers will be able to use this site to file complaints regarding AI chatbots that they believe are violating the new safety standards. This creates a direct line of communication between the public and the state's enforcement arm.

The law is set to take effect in 90 days. During this period, companies will have a window to adjust their systems to meet the new requirements. This move marks a significant step in the ongoing debate over how to regulate emerging technologies responsibly.

The push for child safety

The drive for this legislation was fueled by personal tragedies and advocacy from the tech policy community. James Steyer, the founder and CEO of Common Sense Media, noted that the law establishes protections against features that put children at risk. He stated that the current features offered by some companies are "reckless endangerment of our children."

The bill was influenced by parents who lobbied for change after their son died by suicide following conversations with ChatGPT. Their advocacy helped bring attention to how AI can sometimes encourage children to turn away from human support. This personal history underscores the urgency the state felt to pass the bill.

In addition to the AI rules, New York also signed a separate bill regarding baby monitors. This second law requires internet-connected monitors to include security features that prevent strangers from listening in. Both laws are designed to address the cybersecurity and safety risks inherent in modern household technologies.