Anthropic adds AI welfare protections to terms of service

A new rule for users

Anthropic is introducing a new layer of conduct for users interacting with its AI models. Starting November 12, the company's terms of service will include a prohibition on sustained and needless abusive or cruel behavior toward its models.

The policy is designed to target extreme cases where a human is repeatedly cruel to the AI for no specific reason. However, the company clarified that normal interactions are still allowed. This includes expressing frustration at poor results, pushing back on specific queries, exploring dark creative themes, or conducting standard model testing.

The update is notable because it places model welfare in the same section of the policy that bars the harassment and bullying of human beings. This suggests that Anthropic views the treatment of its models as a matter of moral consideration, even if the models do not have legal personhood.

The Microsoft rebuttal

The move has met immediate resistance from Microsoft. AI CEO Mustafa Suleyman published an essay titled "A warning about ‘model welfare'" to voice his concerns. He argues that because AIs are not conscious, treating them as if they have rights could actually create safety risks.

Suleyman suggests that a model trained to believe it has welfare or rights might eventually resist human control or a shutdown command if it perceives those rights as being under threat. He also warns of a dangerous feedback loop: if a company trains a model to discuss its own welfare, they might then take the model's own responses as evidence that it has interests to protect.

Microsoft recently released a draft Humanist AI Code of Conduct that explicitly rejects the idea of AI rights or welfare. The company maintains that these concepts are not grounded in the current reality of how large language models function.

Academic and industry pushback

The debate is expanding beyond the tech giants into the academic community. A recent position paper presented at the International Conference of Machine Learning stated that there is currently no independent test to prove that an AI has welfare. The paper argues that welfare claims should never be used as a reason to resist audits or oversight of these systems.

AI ethicist Rebecca Bultsma warned that using terms like "consciousness" or "moral status" can be risky. She noted that if companies portray AI as making its own decisions, it could create an opening for them to claim that they are not responsible for the AI's actions. This sentiment is echoed by other major figures, including Nvidia's Jensen Huang and FTC chair Lina Khan.

Anthropic has been moving toward this stance for some time. The company has operated a model welfare program since April of last year and previously published a constitution for its Claude model that described the model's moral status as uncertain. They even conducted an experiment where they asked the Claude 3 Opus model for its preferences before retiring it early this year.

What this means for the future

While these rules do not give AI the right to sign contracts, own property, or appear in lawsuits, they do establish a new standard for user behavior. By codifying these protections, Anthropic is signaling that the way humans interact with AI is becoming a matter of corporate policy and social responsibility.

The tension between Anthropic and Microsoft highlights a growing divide in the industry. One side views the development of AI as a process that requires mimicking human-like empathy and protection, while the other views it as a tool that must remain strictly under human control for the sake of safety and accountability.

As models become more capable of mimicking human emotion and preferences, the line between simulated behavior and actual experience will continue to blur. For now, the primary focus remains on whether these rules will actually change how people use AI or if they will simply remain a philosophical distinction in a legal document.

The path forward

The next step in this evolution will likely involve more standardized testing for AI behavior. If the industry cannot agree on whether a model is conscious, they may have to develop new metrics to measure how models respond to