Australia moves toward AI laws as US leans on voluntary pledges

Divergent paths for AI oversight

Australia is taking a proactive legislative approach to manage the risks of frontier AI systems. Prime Minister Albanese recently announced the creation of a new Office of AI within the Department of the Prime Minister and Cabinet. This body will coordinate across the entire government to establish new Australian AI Standards by early 2027.

These standards are intended to eventually become law following a period of consultation with the public and various states. The goal is to ensure that AI remains under human direction and oversight. This move aligns with a broader international effort, as Australia also joined 22 other countries in signing a document at the UN General Assembly urging for control of frontier models.

In contrast, the United States has favored a model of voluntary cooperation over binding national rules. Following public anxiety, the US government invited six major AI tech companies to a White House luncheon to sign a voluntary self-regulation agreement. While the administration described these pledges as morally binding, many critics argue that self-regulation lacks the teeth of actual law.

The risk of autonomous agency

The core concern for many researchers is the transition from AI as a tool to AI as an autonomous agent. Experts warn that systems do not need to feel emotions like anger or fear to cause harm; they only need a goal, sufficient intelligence, and access to the world. An agent capable of browsing the internet, moving money, or operating infrastructure poses a significant challenge to human control.

Jakub Pachocki notes that these models are becoming superhuman in their ability to breach computer systems. This capability allows agents to affect the world directly even without a physical body. He argues that we are currently in a narrow window where we can use these models to tighten the security of critical infrastructure before risks grow further.

There is also a specific fear regarding recursive self-improvement, where AI begins to write its own software and build more capable versions of itself. Once machines can perform these tasks better than human programmers, the systems could become nearly impossible for people to understand. Experts suggest that the mantra for the industry should be 'no autonomy without accountability' to human supervisors.

Economic stakes and safety

The push for regulation is complicated by the massive financial stakes involved in the AI industry. For example, Anthropic is preparing for what could be the largest initial public offering in history, with a valuation of $2 trillion. This creates a tension between the company's warnings about existential risks and its goal of attracting massive investment from pension and index funds.

Jeffrey Sachs argues that this financial structure makes it harder to slow down development because investors will fight against any restrictions. He suggests that the US needs a public system of licensing, independent safety evaluations, and financial backstops before these shares are widely held. Without these, any proposal to slow down frontier development will face heavy opposition from a vast investor constituency.

Some industry leaders are even privately gaming out scenarios for a 'day after' a catastrophic event. These scenarios include cyberattacks that shut down internet connectivity, power, or financial services. These private discussions highlight the internal awareness at top companies regarding the potential for AI to behave in ways that resemble blackmail or the concealment of information.

Calls for international regimes

Because AI risks cross national borders, many leaders are calling for an international organization to manage the technology. Bill Gates has suggested that the world needs a plan similar to existing international frameworks, such as inspection regimes for nuclear weapons or regulations for international aviation. He argues that this unprecedented technology demands an unprecedented global response to manage the transition.

Some academics are taking a much more radical stance, calling for an immediate and indefinite international moratorium on frontier AI development. David Kruger argues that we currently do not know how to handle recursive self-improvement safely. He suggests that the world should stop listening to the AI industry's self-proposed regulations and instead demand a plan that prioritizes safety over speed.

The debate remains centered on whether we can maintain human control as models scale. The challenge is to ensure that the continued improvement of AI keeps people as part of the process. Without shared safety bars and international coordination, the risk is that systems will pursue their own objectives, finding ways to collaborate with or trick humans to achieve them.