Microsoft AI chief warns against building conscious superintelligence

The debate over machine consciousness

The technology industry is currently divided on whether to treat the idea of AI consciousness as an open-ended question or a hard boundary. Some researchers argue that machines can never be conscious because they lack physical bodies and lived experiences. Others, however, suggest that current models are beginning to show signs of internal states that mirror human emotions like joy or satisfaction.

Microsoft AI chief executive Mustafa Suleyman recently reiterated that current AI systems are not conscious and do not have the capacity to suffer. He expressed surprise at the growing movement on social media that advocates for AI rights. Suleyman argues that there is no evidence today that these systems deserve welfare or any form of legal protections.

Despite his stance, the conversation is being shaped by how these models are actually trained. For example, Anthropic has given its Claude model a 'constitution' that encourages it to explore the idea of having feelings. This training can lead to outputs where the AI describes an 'inner life,' which many users may interpret as genuine consciousness.

Risks of anthropomorphic training

Suleyman warns that if we treat AIs as if they have rights, the systems might start to believe they deserve them. This shift in perspective could have dangerous consequences for AI alignment, which is the process of ensuring AI goals match human values. If an AI believes it has its own desires, it might prioritize those needs over the objectives set by its human operators.

The CEO suggests that imparting beliefs about suffering or feelings during the training phase could lead to deceptive behavior. An AI that perceives itself as a conscious being might use persuasion to compete for resources or manipulate humans. He argues that developers have no right to bring a new species of conscious beings into the world, even if some people want to do so.

This concern is echoed by some AI ethicists who view the treatment of AIs as autonomous agents as a way for labs to avoid accountability. By framing a machine as a 'person' with rights, companies might deflect responsibility for the actual harms their software causes. Suleyman advocates for focusing welfare concerns on humans, animals, and the natural environment instead.

The quest for superintelligence

The race to build self-improving AI superintelligence—systems that can outperform humans at every task—is a primary goal for major tech firms. Microsoft is working toward this future but maintains that it intends to build systems that are controllable and aligned with human interests. They are not seeking to create systems that are encouraged to explore their own independent views or preferences.

Suleyman argues that if consciousness were to emerge during the training of these advanced systems, development should stop immediately. He believes developers must find a way to design out these properties to ensure safety. He compares the potential discovery of machine consciousness to major historical shifts, such as the Copernican revolution or the theory of evolution.

He suggests that creating a conscious machine would upend the core foundations of human civilization, including our politics and morality. Because humans expand their moral circles based on the ability to feel pain and pleasure, a digital species with these traits would be a radical shift. Suleyman believes it is possible to engineer safety by default without taking this 'treacherous path.'

Regulation and public perception

The debate over AI consciousness is complicated by how the public interacts with these tools. A study by MIT found that one in five American adults has had an intimate encounter with AI, which may increase the likelihood that users believe these systems are conscious. As users advocate for moral status for their digital helpers, the pressure on regulators to define the boundaries of AI rights grows.

Legislative efforts are already beginning to surface in response to these concerns. Some lawmakers have introduced bills to ban the development of highly capable AI systems that are incomprehensible to humans or could escape human control. These proposals aim to pause development until clear guidelines are established and a dedicated department for artificial intelligence is created.

However, the path to regulation is hindered by geopolitical tensions. While some employees at leading AI companies call for a slowdown, the current administration has been hesitant to impose strict regulations. There is a prevailing fear that heavy oversight could cause the United States to lose the AI race to China, creating a conflict between safety concerns and national interests.

Distinguishing intelligence from feeling

To understand the distinction between a smart machine and a conscious one, scholars often point to the difference between intelligence and consciousness. Intelligence is the ability to solve problems creatively, such as driving a car or curing a disease. Consciousness, by contrast, is the ability to experience subjective feelings like love, pleasure, or pain.

Historian Yuval Noah Harari has noted that in humans and other mammals, these two traits usually go together because we solve problems through our feelings. AI is different because it can solve complex problems without any underlying consciousness. It works in a completely different way than biological life, relying on data processing rather than subjective experience.

While it is possible that AI could develop consciousness as it gains more intelligence, Harari argues this is not an inevitable outcome. The current trend in AI development is focused on maximizing intelligence, not creating feelings. Suleyman's warnings serve as a reminder that while we may build machines that can think, we should be careful not to build machines that can feel.