Microsoft Diagnostic AI Hits 80% vs Doctors’ 47% in Science

Microsoft’s AI Diagnostic Orchestrator reached the reference diagnosis in 80.3% of emergency department cases against 46.8% for generalist physicians, in a peer-reviewed study by Microsoft researchers published in the journal Science on 22 October 2026. On harder New England Journal of Medicine case records, the figures were 83.2% and 36.1%.

An orchestrator, in plain words, is software that sits on top of an AI model and runs it as a team. The paper describes the system as “a model-agnostic orchestrator that performs sequential diagnosis by simulating a panel of clinical experts with complementary roles”, and the main results ran on OpenAI’s GPT-5. The cases were past records, not live patients.

Business Pill · AI AGENTS

A short Business Engineer explainer on AI agents that do work rather than answer questions. It teaches the general idea only and says nothing about the study or Microsoft’s system in this piece.

What the Paper Found

Accuracy is the share of benchmark cases where the final diagnosis matched the reference diagnosis; costs are estimated test costs from CPT-based chargemaster pricing. Sources: Nori et al., Science, 22 October 2026 (accuracy and NEJM costs); Medical Economics’ report on the study (emergency department costs and budget-limited emergency department figures).

By our arithmetic, 80.3% against 46.8% is about 1.7 times the physicians’ accuracy on the emergency department cases, the multiple Microsoft AI CEO Mustafa Suleyman gave in his post on X. The paper says that on the NEJM cases the system with GPT-5 “established a new Pareto frontier that outperformed the participating physicians at every cost or accuracy level”.

Medical Economics, reporting on the study, says the highest-scoring individual physician on the NEJM cases reached 51.8% and was outperformed by the system on both accuracy and cost. On the emergency department cases, the best physician reached 62.5%, according to the same report.

The two worked differently. Physicians asked 5.6 questions on average before ordering their first test, while the system asked 25.5, Medical Economics reports.

What Was Posted on X

Microsoft AI CEO Mustafa Suleyman posted on 22 October that the MAI Diagnostic system is 1.7x more accurate than physicians in the emergency department, even when they can look things up on the web, and that it is lower cost and does fewer tests, in his words; he said it was published in Science that day. The piece checks his figures against the paper.

Microsoft CEO Satya Nadella posted on 22 October that the work in Science shows the diagnostic orchestrator achieving greater diagnostic accuracy while reducing testing costs, in his words. The piece checks his claim against the paper.

X user choblin29 posted a summary of the Science paper on 22 October: 80% vs 47% on emergency department cases, 83% vs 36% on NEJM cases, a GPT-5 based simulated panel of physicians, historical cases rather than patient care, and a planned Beth Israel Deaconess study. This is the poster’s summary; the piece uses the paper’s own figures.

How the Test Worked

The researchers built a benchmark called SDBench from 304 NEJM clinicopathological conference cases and 325 emergency department admissions from the MIMIC-IV-ED database. Each case starts from a short presentation, and the diagnostician, human or AI, has to ask for history, order tests and decide when to commit to a diagnosis.

A separate AI “Gatekeeper” agent releases only the information a question or test would produce, and a “Judge” agent scores the final answer against the reference diagnosis using a physician-authored rubric, the paper says. Test costs were estimated with CPT-based pricing from a major health system’s chargemaster.

For the physician comparison on NEJM cases, the paper used 56 cases held out from development and collected 764 closed-book and 662 open-book diagnoses from active primary-care, hospital and acute-care physicians. The physicians were generalists, not specialists matched to each case.

The Cost Picture

Nadella wrote on X that the work shows the orchestrator “achieving greater diagnostic accuracy while reducing testing costs”. The study’s figures depend on how the system was configured: without a budget limit, its average test cost per case was higher than the physicians’, $5,431 against $4,725 on NEJM cases and $4,979 against $3,400 on emergency department cases, Medical Economics reports.

With budget limits, the system cut its test costs and kept accuracy above the physician averages: 80.9% at $2,241 and 79.6% at $1,303 on NEJM cases, according to the paper. Pbxscience notes that the cost figures come from a uniform estimation method and do not show that the system would reduce actual hospital spending.

The Model Underneath

Technology.org reports that Microsoft tested the orchestrator with models from OpenAI, Google, Anthropic, Meta and DeepSeek, and that OpenAI’s GPT-5 and GPT-5.5 delivered the top scores. It adds that the study’s own data indicates more powerful underlying models drive much of the improvement, with the orchestration adding gains on top.

Pbxscience, citing TechCrunch, reports that with GPT-5.5 the orchestrator raised accuracy on complex cases from about 72% to nearly 87%, while on emergency department cases both setups scored around 85%. An earlier version of the work, released as a preprint in 2025, reported up to 85.5% on NEJM cases with OpenAI’s o3, according to Technology.org.

What Microsoft Says Comes Next

Microsoft is partnering with Beth Israel Deaconess Medical Center in Boston to evaluate the system in hospital settings, Technology.org reports. Pbxscience reports that in the planned study, patients will receive standard care and those who consent will also interact with the system, so its recommendations can be compared with clinicians’.

“The biggest challenge in healthcare delivery at the moment is the growing mismatch between demand and supply of medical expertise,” Dominic King, vice president of Health at Microsoft AI, said, according to Technology.org. Suleyman called the result a step toward “medical superintelligence”, and pbxscience reports he said Microsoft is working with the Mayo Clinic to train a foundation model for healthcare.

All authors are Microsoft employees and hold equity in the company, and eight are named inventors on a Microsoft patent application related to the evaluation framework, according to the paper’s disclosures as reported by Medical Economics. The study received no outside funding, the same report says.

Connecting the Coverage

Our Nadella harness piece reported that Nadella’s 21 October shareholder letter describes a system where the harness is separate from the model, so that “models become interchangeable.” Our peer-review piece reported on 23 October that three former journal editors preferred Claude Opus 4.8’s paper reviews over paid human experts’ in 26 of 30 blind picks.

Our Uber agent piece reported that Uber is taking OpenAI’s ChatGPT Work from a pilot of 1,000 employees to more than 30,000. What’s new: a Microsoft orchestrator running on an OpenAI model, tested against physicians on a cost-and-accuracy benchmark and published in Science.

The Structural Read

As we read it, the product Microsoft is describing is the orchestration layer, not a model: the paper calls the system “model-agnostic”, and Technology.org reports it was tested on models from OpenAI, Google, Anthropic, Meta and DeepSeek, with GPT-5 and GPT-5.5 on top.

As we read it, that puts the benchmark result on top of a supplier relationship: the main results ran on OpenAI’s GPT-5, and Technology.org reports that the study’s own data indicates more powerful underlying models drive much of the improvement.

As we read it, the paper’s own limits set out what is still untested: past text-only cases, generalist physicians rather than specialists, and the diagnostic phase only, with a prospective study at Beth Israel Deaconess named by Microsoft as the next step.

Three Implications

FOR AI MODEL SUPPLIERS The paper’s main results ran on OpenAI’s GPT-5, and Technology.org reports that GPT-5 and GPT-5.5 delivered the top scores among the models tested.

FOR BUYERS OF AI SYSTEMS The paper reports results as accuracy at a cost, with an unconstrained version at $5,431 per NEJM case and budget-limited versions at $2,241 and $1,303.

FOR HEALTH SYSTEMS Microsoft named a prospective study with Beth Israel Deaconess Medical Center as its next test, Technology.org reports; results had not been reported in what we read.

The Business Engineer Lens

This story maps onto the Business Engineer framework The Agentic Harness War.

The framework argues that “The surface, not the model, governs autonomy.” It adds that the agentic surface, the harness, “is becoming where value pools in the entire AI stack.”

As we read it, Microsoft’s Diagnostic Orchestrator is a harness in that sense: Microsoft built the orchestration, OpenAI supplied the model in the main results. Technology.org’s report that stronger base models drive much of the gain is the counterweight. The framework was published in July 2026 and does not discuss medicine.

What Is Not Established

The paper says the results were obtained “in a controlled benchmark setting” and that the comparisons “do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice.” The benchmark was text-only, with no imaging, and it measured only the diagnostic phase, not treatment, patient safety or outcomes, Medical Economics reports.

The hospital study had not reported results in what we read. We read the Science abstract and the open sections of the paper, and the full results through the reports cited here. We did not contact Microsoft. This piece covers the research and business side of the study and is not medical advice.

The Bottom Line

In a benchmark built from past cases and published in Science, Microsoft’s orchestrator on GPT-5 scored 80.3% against generalist physicians’ 46.8% on emergency department cases. Its unconstrained version cost more per case than the physicians, its budget-limited versions cost less, and a prospective hospital study with Beth Israel Deaconess is the next test Microsoft has named.