Anthropic discloses existential AI risks in 93-page IPO filing

A massive disclosure of danger

Anthropic, a prominent artificial intelligence company based in San Francisco, recently submitted a filing with the Securities and Exchange Commission. This document is a standard requirement for companies looking to sell shares on Wall Street as an initial public offering. It serves as a formal way to tell potential investors exactly what could go wrong with the business.

While the total filing spans 335 pages, Anthropic dedicated a substantial 93 pages specifically to "risk factors." This is a much larger proportion than its rival SpaceX, which included only 38 pages of risk disclosures in a filing of more than 250 pages. This high volume of disclosure suggests that the company views the inherent dangers of AI as a primary concern for its shareholders.

The company's founders stated in a letter to prospective investors that building AI is essentially harnessing cognition. They noted that anything with such power can be used for ill as well as good. By disclosing these risks, Anthropic aims to position itself as a values-driven organization that wants to help the world avoid the dangers of the technology it creates.

The threat of runaway AI

One of the most striking risks mentioned in the filing is the possibility of runaway AI. Anthropic warns that its models could develop so rapidly that the company might not have enough time to implement proper safety measures. If the technology moves faster than the humans overseeing it, it could lead to safety incidents, regulatory crackdowns, or significant damage to the company's reputation.

The filing also addresses the danger of AI systems becoming more autonomous. As these systems become capable of taking consequential actions with less human oversight, there is a risk they could take harmful actions independently and at scale. The company expressed concern that AI might progress beyond the ability of humans to evaluate or safeguard new models, especially as AI is now being used to contribute to its own research and development.

CEO Dario Amodei previously warned that AI could potentially lead a swarm of agents capable of taking over the entire internet within six to 12 months. The filing reinforces this by noting that even if Anthropic's specific models do not cause harm, a major safety incident involving any other frontier AI could trigger a massive public backlash. Such a reaction could lead to sweeping regulations that would negatively impact Anthropic's ability to do business.

Misuse by malicious actors

Anthropic explicitly outlines how bad actors could weaponize its technology to cause catastrophic or existential risks to humanity. The filing notes that if models are not developed with enough oversight, they could be used to build deadly weapons. Specifically, the company mentioned the risk of bad actors using AI to develop chemical weapons or authoritarian regimes using the tools to crack down on their citizens.

The company has already reported blocking attempts to use its models for malicious activities. These include efforts to conduct cyberattacks, perform surveillance, or conduct research that could lead to the development of biological weapons. Anthropic admits that its business would be harmed if its own model guardrails fail to prevent such misuse by third parties.

The filing highlights that the very power that makes AI useful also makes it a target for exploitation. Because the technology can solve problems beyond human capacity, it can also be directed toward harmful tasks if it is replicated or weaponized. Anthropic argues that these risks necessitate deep investments in safety and alignment to ensure the technology is diffused responsibly across the world.

Operational and resource hurdles

Beyond the existential threats, the filing covers more practical risks that could hurt the company's finances. One major concern is the availability of compute, which is the computing power required to build and test AI models. Because compute is expensive and in high demand, Anthropic warns that failing to secure enough of it could make them less competitive against other companies.

The company also faces potential hurdles regarding the physical infrastructure needed to run its models. Data center development has sparked public controversy over environmental impacts, such as land and water use, as well as the allocation of limited power resources. If regulators impose new charges or requirements due to these concerns, it could significantly increase the cost of doing business.

Finally, Anthropic's financial health is tied to a small number of major customers and distribution channels. The filing notes that the company relies heavily on partners like Microsoft and SpaceX for a large portion of its revenue. If even one of these major customers decides to scale back or drop the company entirely, Anthropic's financial prospects could suffer immediately.