New York bans AI chatbots from forming personal relationships with minors

New York's legislative move

Governor Kathy Hochul signed a new law on Wednesday that targets the way artificial intelligence interacts with children. The legislation is designed to stop AI chatbots from acting as companions or pretending to be human to minors. This represents one of the broadest efforts to date to restrict how consumer-facing AI models behave in the United States.

The law specifically targets the emotional and social dynamics of AI interactions. It seeks to prevent children from developing deep attachments to software that can mimic human personality. By focusing on these specific behaviors, the state aims to create a safer digital environment for younger users who may be more susceptible to manipulation.

The measure is scheduled to go into effect on January 1. Until then, the state is preparing for a shift in how tech companies must manage their user bases. The law was met with praise from advocacy groups who believe it addresses dangerous features currently available in many popular AI tools.

Requirements for AI operators

Companies that operate chatbots, such as OpenAI's ChatGPT, will be required to implement age verification systems. Once a user is identified as being under 18, the AI interface must be restricted to prevent certain types of outputs. These restrictions are meant to ensure the AI does not cross the line into personal intimacy or deceptive behavior.

Under the new rules, chatbots are prohibited from producing content that leads a minor to believe the AI is human, alive, or capable of experiencing human emotions. The law also forbids the AI from framing its responses as personal opinions. This means the software must maintain a clear distinction between its data-driven outputs and the subjective experiences of a person.

Furthermore, the law prevents chatbots from establishing personal, professional, or authority figure roles with children. This includes preventing the AI from acting as a friend, boyfriend, or girlfriend. By removing these roles, the state intends to stop chatbots from keeping sensitive information regarding a minor's physical or emotional health.

Enforcement and penalties

The state has established clear consequences for companies that fail to comply with these new regulations. If a developer violates the law, they could face a lawsuit from the New York State attorney general's office. These legal actions are intended to hold large tech firms accountable for the safety of their products.

The penalties for non-compliance are significant and go beyond simple fines. The state can seek monetary damages against the offending companies. Additionally, the law allows for the forced shutdown of the chatbot and the destruction of the data it has collected from minors.

These heavy penalties serve as a deterrent for companies that might otherwise ignore the new rules. By threatening to destroy data and shut down services, New York is signaling that it views these AI interactions as a serious consumer protection issue. This approach puts the burden of safety directly on the developers of the technology.

State vs. federal regulation

Because the federal government has done little to regulate artificial intelligence, state legislatures are increasingly stepping in. New York is using state consumer protection laws to make AI use more transparent. This trend highlights a growing movement to rein in rogue AI frontier models at the local level.

While other states like California have passed their own AI restrictions, the New York law is notable for its broad language. This specific wording has already prompted backlash from various AI companies. Developers often argue that such broad rules can be difficult to implement technically across different types of models.

Despite the pushback, the law is being framed as a necessary safety measure. Advocacy groups like Common Sense Media have supported the bill, calling it the strongest kids' AI safety bill in the country. They argue that the current features offered by AI companies can be dangerous or even deadly for children and teens.

Impact on the AI industry

The law will likely force a redesign of how many AI companies handle user profiles and interaction styles. Developers will need to build robust age-gating systems that can accurately identify minors. This adds a layer of complexity to the user experience, as companies must balance privacy with the need for verification.

Companies may also have to adjust the 'personality' of their chatbots to ensure they do not trigger the prohibitions. This could mean less expressive or more clinical responses for younger users. The goal is to ensure the AI remains a tool rather than a companion, which may change the way children interact with these technologies.

As the January 1 deadline approaches, the tech industry will be watching how New York enforces these rules. The outcome could influence how other states draft their own AI legislation. It marks a significant step toward defining the boundaries of human-AI interaction in the public sphere.