Anthropic staff warn of AI risks as company softens safety rules

Internal warnings of existential risk

Jacob Coxon, a researcher who spent three years at both OpenAI and Anthropic, recently resigned from the latter company. He publicly stated that the people building AI earnestly believe the technology could kill everyone by the end of the decade. Coxon argued that the companies are racing toward self-improving superintelligence while gambling with human lives.

His concerns were echoed by Evan Hubinger, who leads the alignment science team at Anthropic. Hubinger agreed with the sentiment that AI poses a unique level of danger compared to any other human activity. He personally estimated that there is a greater than 10% chance that AI could kill all humans within the next ten years.

Even Anthropic's leadership has acknowledged these potential dangers in public forums. CEO Dario Amodei stated at a summit in September 2025 that there is a 25% chance the future of AI goes 'really, really badly.' He noted that the company feels the clock ticking down as the technology becomes more advanced.

The weakening of safety guardrails

Anthropic previously maintained a scaling policy promising to stop training and releasing any model it could not make safe. However, the company softened this language on Feb. 24 to commit to delays only in narrow cases. The company argues that stopping on its own would simply hand the lead to rivals with weaker protections.

Critics from the research group GovAI expressed concern that this shift makes it more likely for Anthropic to deploy models with unacceptable risks. They noted that the lines for triggering the strongest protections keep moving as models climb toward them. The company has already revised its protocols twice since Feb. 10 to accommodate these shifting boundaries.

The new bar for intervention now measures how much AI is speeding up Anthropic's own internal research. The company admits that ruling out models is becoming increasingly difficult and often relies on subjective judgments. While they recommend strict safeguards for the industry, they stated they cannot commit to following them on their own.

Filtering biological and chemical threats

To manage immediate risks, Anthropic uses classifiers, which are separate AI models that monitor conversations in real time. These filters are designed to block a narrow set of requests related to building weapons. The company turned these protections on in May 2025 with the launch of Claude Opus 4.

The filters are becoming more critical because newer models are more capable of assisting with dangerous tasks. Anthropic noted in a threat report that older models were well below the threshold for meaningfully assisting in bioweapons development. They stated that this is no longer a certainty with their newer, more powerful models.

The company has already caught several researchers attempting to use the AI for dangerous purposes. These included requests to draft proposals for mutating the chikungunya virus and work on making bird flu spread more easily in mammals. Anthropic blocked these requests and, in some cases, turned the information over to law enforcement.

External reviews and industry pressure

Despite Anthropic's internal safety efforts, outside reviewers have expressed significant skepticism about the company's plans. In a 2026 AI Safety Index, the Future of Life Institute gave Anthropic a C+ grade, the highest among the nine companies reviewed. The report stated that even the best performers remain far from meeting adequate safety expectations.

Reviewers were particularly critical of the company's lack of an explicit safety strategy for keeping control of far more capable AI. One reviewer described Anthropic's retreat on its safety framework as undermining safety standards across the entire industry. They recommended that the company return to its original promise to stop training any model it cannot make safe.

Dario Amodei has called for the entire industry to slow down to prevent AI from potentially taking over the internet in as little as six to 12 months. However, despite these calls, every top AI company is still building at the same pace. This leaves Anthropic largely responsible for checking its own work, as its CEO and safety officer provide the final sign-offs.