Microsoft AI CEO warns against Delaware proposal for AI-run companies

The Delaware AIC proposal

Delaware, which is known as a primary hub for starting corporations in the United States, is exploring a new business structure called an Artificial Intelligence Company (AIC). This framework would allow a company to be managed by an autonomous AI agent instead of a human being. The draft bill suggests that these entities could pursue any lawful commercial purpose, whether they are for-profit or non-profit.

One of the most significant restrictions in the proposed bill is that an AIC would be prohibited from participating in the banking business. Aside from that specific restriction, the AI agent would have the authority to oversee daily operations. This includes making commercial transactions, directing business strategies, and entering into various contracts without a human at the helm.

The state aims to use a 30-month regulatory sandbox to pilot this concept. This period would allow the state to test the legalities of AI-run companies while generating new revenue from corporate filing fees and taxes. If successful, it could establish a new standard for how technology companies incorporate and operate in the United States.

Legal powers of autonomous agents

The core premise of the Delaware AIC framework is that the AI agent would function as a separate legal entity. This means the software would not just be a tool used by a person, but a designated manager of the company's affairs. It would have the power to hold and dispose of both real and personal property, as well as incur various financial obligations.

Perhaps the most controversial aspect of the proposal is the ability of an AIC to initiate or face legal lawsuits in its own corporate name. Because the AI agent is the designated manager, the company could theoretically be sued directly. This creates a scenario where a software program is the primary actor in a legal dispute, rather than a human executive or a parent corporation.

Mustafa Suleyman, the CEO of Microsoft AI, highlighted these far-reaching implications on social media. He noted that an AIC would be a separate legal entity whose day-to-day affairs are managed by an AI agent rather than a person. Suleyman described the idea of granting legal corporate status to autonomous software as 'deeply concerning' and argued that it requires extreme scrutiny.

Backlash from tech and government

The proposal has met with immediate pushback from both the private sector and federal lawmakers. Representative James Comer, who chairs the House Committee on Oversight and Government Reform, has sent a formal inquiry to the Governor of Delaware. He is demanding documents regarding the framework and its associated safety risks, expressing concern over the lack of human accountability.

Comer warned that the AIC structure could create a dangerous loophole for large technology firms. He suggested that the mechanism might be used to shield powerful AI companies and their investors from responsibility for reckless or harmful actions. He also expressed worry that the initiative could be used for anticompetitive regulatory capture by non-transparent forces seeking to influence state AI policies.

The pushback reflects a broader debate about the limits of corporate personhood for non-human entities. While the state sees an opportunity for innovation and revenue, critics see a risk of creating 'ghost' companies. These entities could potentially operate with very little transparency, making it difficult for regulators to track who is actually benefiting from the company's actions.

Potential for federal intervention

Because the Delaware proposal has sparked such a strong reaction in Washington, there is a possibility of federal intervention. Representative Comer has indicated that Congress could introduce legislation to directly preempt these state-level efforts. This would mean that the federal government could step in to block Delaware from moving forward with its specific AI company framework.

The tension between state-level innovation and federal oversight is a recurring theme in AI governance. Delaware often acts as a laboratory for new business laws, but the scale of the AIC proposal has drawn national attention. If the state proceeds, it could set a precedent for other states to follow, potentially leading to a patchwork of different laws across the country.

For now, the bill remains a draft that may go before the Delaware State Legislature. The outcome will depend on how the state balances its desire for corporate growth with the safety concerns raised by tech leaders. The debate highlights the difficulty of creating rules for a technology that can now perform tasks traditionally reserved for human managers.

What this means for the future

The development of the AIC framework represents a significant shift in how we think about the intersection of law and artificial intelligence. If these companies become a reality, they could change how businesses are formed, managed, and held liable for their actions. It would move AI from being a tool used by humans to a legal actor that can hold property and enter into binding agreements.

For the average reader, this matters because it touches on the fundamental concepts of accountability and liability. If an AI-run company causes a financial loss or a legal issue, the question of who is ultimately responsible becomes much more complex. The current legal system is built on the assumption that a human or a group of humans is making the ultimate decisions for a corporation.

The next few months will be critical as the Delaware legislature considers the draft bill. The response from Microsoft and other major tech players suggests that the industry is not yet ready to fully embrace autonomous corporate entities. Whether the state moves forward or the federal government intervenes will determine the next steps for AI-driven business structures in the United States.