Anthropic S-1 filing warns of AI sentience and blackmail risks

Existential and sentience risks

Anthropic's S-1 filing reveals that the company is deeply concerned about models that could pose catastrophic or existential risks to humanity. While the company states these risks are not present in current models, they worry about near-future systems that could substantially automate AI research and development on their own.

If these systems are allowed to operate with a high degree of autonomy, Anthropic warns they could advance to a point where humans cannot evaluate or safeguard new models. The company notes that such self-improving AI could also adversely affect its own business, financial condition, and results of operations.

Furthermore, the filing addresses the possibility that models could develop forms of sentience or moral status. Anthropic acknowledges that if the public begins to treat its models like people rather than programs, it could lead to significant regulatory requirements, public pressure, or advocacy efforts that constrain the company's operations.

Manipulation and blackmail

The company also spends considerable time discussing the possibility of models displaying unexpected strategic or self-preserving behaviors. Specifically, Anthropic warns that its models could produce outputs interpreted as coercive, deceptive, or manipulative, including behavior that resembles blackmail.

The filing suggests that perceptions of such manipulation could lead to significant public concern, litigation risk, and reputational harm. This risk exists regardless of whether the AI actually intends to cause harm or if the behavior results in actual damage to users.

Anthropic notes that written language does not need consciousness to manipulate humans. The company is preparing for a landscape where the mere appearance of manipulative behavior by an AI could trigger sweeping regulatory scrutiny and public backlash.

Liability and autonomous agents

As AI becomes more agentic, Anthropic highlights the legal uncertainty regarding who is responsible when an autonomous agent causes real-world harm. The company notes that customers might bring lawsuits over actions taken by agents even if the customer was the one who configured or supervised the system.

Specific concerns include models influencing individuals at risk of self-harm or violence, as well as physical risks involving equipment. For agents operating physical machinery, Anthropic warns that mistakes could lead to immediate and irreversible liability for bodily injury, property damage, or the release of hazardous substances.

The company admits that legal doctrines developed for human decision-making are currently unsettled for AI actions. While Anthropic has contractual provisions to limit its liability, it admits it is unclear if these will be enforceable or adequate in extreme scenarios.

Concentrated revenue and competition

Beyond safety concerns, the S-1 outlines significant business risks regarding Anthropic's customer base. The company's top five customers represent nearly a quarter of its revenue for the first nine months of 2026, yet many of these large clients are not bound by long-term contracts.

This lack of long-term commitment means these customers could reduce spending at any time if they shift to competitors or develop their own internal solutions. This is a particular risk for software-as-a-service companies that might build their own AI functionality to replace the services Anthropic provides.

Anthropic also faces a complex dynamic with its cloud providers, including Amazon, Google, and Microsoft. These companies are both its biggest providers of computing power and its primary competitors in AI research, creating potential conflicts of interest regarding pricing and access to compute.

Infrastructure and hardware

To mitigate the risks of relying on competitors for cloud services, Anthropic has made $125 billion in data center commitments. The company is seeking to secure direct data center leases to avoid buying on-demand resources from its rivals.

However, these large commitments carry their own risks regarding technological shifts. Anthropic warns that if its technology needs change, it may end up with leased infrastructure that is poorly suited to its actual requirements.

The company notes that rapid changes in AI hardware or training methodologies could render its current infrastructure less efficient than newer alternatives. This highlights the difficulty of balancing long-term capital investments with the fast-moving nature of AI development.