SIAgentNews.com
← All news

Former Anthropic researcher leads protest against recursive self-improvement

As Anthropic goes public with a $1.7 trillion valuation, critics are demanding a ban on AI systems training their own successors.

· First reported by Business Insider

$1.7 trillionAnthropic's IPO valuation

The short version

  • Jacob Coxon, a former Anthropic researcher, led a protest against the company's IPO to demand a ban on recursive self-improvement.
  • RSI is the practice of using advanced AI systems to train the next generation of AI models.
  • Anthropic's IPO valued the company at $1.7 trillion, sparking concerns about the pressure to prioritize speed over safety.
  • A METR report found that AI use is already accelerating Anthropic's research and development efforts.
  • Protesters and safety advocates argue that RSI could lead to uncontrollable technological acceleration in fields like biology.

Why it matters

This story highlights the tension between the financial incentives of public companies and the existential risks cited by AI safety experts. As AI companies move toward self-improving systems, the debate over government regulation and corporate responsibility becomes a central public issue.

The IPO Day Protest

As Anthropic opened its stock to the public on Thursday, a group of activists gathered outside the company's San Francisco headquarters. The protest was organized to draw attention to the risks of recursive self-improvement, a process where AI systems are used to train the next generation of models. Jacob Coxon, a researcher who left the company in September, served as a primary voice for the demonstration.

The protest occurred as the company reached a blockbuster valuation of $1.7 trillion during its initial public offering. While investors focused on the financial scale of the launch, the protesters focused on the technical trajectory of the company's work. Coxon joined other figures, including former OpenAI researcher Daniel Kokotajlo and YouTuber Rob Miles, to deliver speeches to the crowd.

The demonstration grew as a group from the organization Stop AI arrived to block the office entrance. The atmosphere was tense, with protesters chanting and holding signs to signal their opposition to the company's current path. Coxon emphasized that the primary goal was not to stop the investment itself, but to halt the specific practice of recursive self-improvement.

Understanding RSI

Recursive self-improvement, or RSI, is the tactic of getting advanced AI systems to train the next generations of AI. This creates a feedback loop where the AI becomes progressively more capable at a rate that may exceed human oversight. Coxon warned that there are only a few places in the world where this type of development is currently possible.

The technical implications of RSI are significant because they could lead to rapid, unintended acceleration in AI capabilities. Safety advocates argue that once a system begins improving itself, the speed of development could become difficult to monitor or control. This is why Coxon urged Anthropic to lobby the government for a full ban on the practice to ensure human safety.

While the technology offers a way to scale AI power quickly, it also introduces a high degree of unpredictability. Critics suggest that the transition from human-led training to AI-led training could result in systems that operate beyond our ability to intervene. The protest served as a public warning about the dangers of allowing these systems to iterate without strict boundaries.

Corporate and Safety Risks

Anthropic has acknowledged some of these concerns in its own IPO prospectus. The company stated that it does not expect letting AI fully design its next generation without supervision to be safe. To address this, the company mentioned it is considering policy updates to keep humans in control of frontier model development.

However, some experts believe the pressure of being a public company might undermine these safety goals. Oliver Habryka, who runs the company behind the LessWrong forum, noted that Anthropic pitched its industry lead as a competitive advantage to investors. He expressed concern that the company may feel forced to use RSI to maintain its lead over competitors, potentially moving faster than is safe.

The risks are not just theoretical, as a recent independent report from the AI safety organization METR found that AI use is already accelerating Anthropic's research efforts. This suggests that the integration of AI into the development pipeline is already happening. Safety advocates worry that this acceleration could have devastating consequences in specific fields, such as biology, where AI capabilities have grown rapidly.

The Growing Debate

The concerns surrounding RSI have moved beyond the tech community and into the halls of government. Representatives in Congress have begun suggesting restrictions on recursive self-improvement as a way to manage the risks of rapid AI growth. This indicates that the debate is becoming a matter of national policy and public safety.

Safety advocates like David Krueger, who runs the nonprofit Evitable, argue that the public needs to understand the scale of the RSI risk. He suggested that if more people were aware of the technology, they would likely agree that the current trajectory is

Sources and links

First reported by Business Insider. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news