Microsoft AI tool outperforms doctors in medical diagnosis tests

Beating human benchmarks

Microsoft recently revealed a new AI-powered tool that it claims can diagnose medical conditions with significantly higher accuracy than human doctors. The findings were published in the journal Science after the company tested the system against various medical scenarios. These tests included data from hundreds of reports in the New England Journal of Medicine and emergency department records from the Beth Israel Deaconess Medical Center in Boston.

The results showed a stark difference in performance between the technology and human practitioners. On the complex cases from medical journals, the AI achieved a diagnostic accuracy of around 83 percent. In contrast, generalist physicians using references scored roughly 36 percent, while those working without references scored only 22 percent.

The system also demonstrated superior performance in emergency department settings. It scored 80 percent accuracy on patient admissions from the Boston hospital, while the average for human physicians was 47 percent. Additionally, the researchers noted that the AI system tended to reach its diagnoses at a lower estimated cost than its human counterparts.

How the orchestrator works

The tool, known as Diagnostic Orchestrator, is designed to function like a panel of medical experts rather than a single software program. It utilizes multiple AI agents that work together to reason through a patient's symptoms and medical history. This collaborative approach allows the system to mimic the investigative process that doctors use to solve complex cases.

The system is powered by OpenAI’s GPT-5 and was originally developed when GPT-4.1 was the most advanced model available. Researchers found that having multiple agents work together provided benefits that were not necessarily diminished by the use of newer, more capable models. This multi-agent architecture helps the AI decide which tests to order and how to weigh different possible explanations for a patient's condition.

To evaluate these capabilities, Microsoft compiled its own benchmark called SDBench. The company intends to open-source this benchmark to provide a standard for measuring how different AI systems perform in medical contexts. By creating a standardized way to test these tools, Microsoft aims to move the industry toward more transparent and comparable results.

The path to clinical use

While the results are impressive, Microsoft acknowledges that the technology is not yet ready for widespread clinical deployment. The current tests were based on past medical cases rather than live patients in a real-time clinical environment. Because of this, the company emphasizes that the system still requires formal clinical validation to ensure it works safely in practice.

Microsoft AI CEO Mustafa Suleyman told The Verge that the company aims to build a system that is indistinguishable from human performance within the next couple of years. He described the current results as a major milestone on a long journey toward what the company calls "medical superintelligence." This vision involves AI that can outperform the best human-led healthcare systems across various categories.

To bridge the gap between research and reality, Microsoft plans to begin testing the system in actual hospital settings. They are partnering with the Beth Israel Deaconess Medical Center to see how the tool performs in real-world environments. Dr. Dominic King, the company's AI health lead, indicated that these tests will likely get underway in the next couple of months.

Impact on healthcare delivery

The ultimate goal for Microsoft is to create an AI that can master several key areas of medicine beyond just diagnosis. These include coordinating clinical care, making treatment decisions, and handling the human side of medicine. The company views these capabilities as essential for changing the way care is delivered on a large scale.

Suleyman maintains a human-centered vision for this technology, insisting that human clinicians will always remain central to patient care. He believes the AI will be most effective at removing the "drudgery" of medicine, which refers to the repetitive and administrative tasks that can burden healthcare workers. By automating these aspects, the goal is to free up doctors to spend more quality time with their patients.

However, the rollout of such technology faces significant hurdles, particularly regarding regulatory and compliance requirements in the medical field. Microsoft has already established a presence in US healthcare through digital tools and its Copilot chatbot. The company is now looking toward a responsible deployment of these more advanced intelligent systems in production environments next year.

What comes next for the tech

The next phase for Microsoft involves moving from retrospective data analysis to live clinical trials. These tests will provide the necessary evidence to determine if the AI can handle the unpredictability of real-time patient interactions. The company is looking for proof that the system can maintain its high accuracy while navigating the complexities of a busy hospital.

Commercialization of the Diagnostic Orchestrator remains a possibility, though Microsoft has not yet announced a specific timeline or method for making it available to the public. The company will likely continue to refine the multi-agent reasoning process to ensure it meets the high standards required for medical safety. They are also monitoring how the system interacts with other frontier AI models as they are released.

As the technology matures, the focus will shift toward how these tools can be integrated into existing hospital workflows. The success of the partnership with Beth Israel Deaconess Medical Center will be a key indicator of whether the system can scale. If successful, it could set the stage for a new standard in how medical professionals access diagnostic support.