🔍 Read the full analysis: What The Study Evaluated In Microsoft’s AI And Doctors Comparison on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A study published in Science on October 22 reported that Microsoft’s MAI-DxO system reached 83.2% diagnostic accuracy on 304 complex historical cases, compared with 36.1% for generalist physicians using reference materials. The test used text-based case records and a simulated process for requesting information; it did not evaluate care delivered to live patients. Microsoft says a prospective study alongside normal care is planned at Beth Israel Deaconess Medical Center.
Microsoft’s MAI-DxO diagnostic AI system scored 83.2% on 304 complex medical case records in a study published October 22 in Science, compared with 36.1% for generalist physicians using textbooks and the internet. The peer-reviewed study evaluated historical, text-only cases in a simulator, not diagnosis or treatment of patients in a clinical setting, limiting what the results establish about hospital use.
The study tested a diagnostic process rather than asking a system to choose an answer from a complete case summary. Each case began with a short presentation. The AI or physician could request additional information, such as a test result, from a simulated gatekeeper holding the full record. The researchers assigned costs to requested tests and assessed both the final diagnosis and the resources used to reach it. Microsoft calls the benchmark the Sequential Diagnosis Benchmark.
Microsoft reported that MAI-DxO reached 80.3% accuracy on 325 emergency department cases, compared with 46.8% for physicians. On the 304 complex cases, the physician score was 22.3% without reference materials and 36.1% with textbooks and internet access. These are results reported by Microsoft’s research team; the study’s publication in a peer-reviewed journal does not itself mean other researchers have reproduced them.
The system uses a language model with an orchestration layer that stages several virtual physician agents. They propose diagnoses, choose follow-up questions or tests, and challenge one another as the case proceeds. In Microsoft’s reported results, GPT-5 alone scored 78.9% on the complex cases. That means the comparison also raises a question about how much of the result comes from the underlying model and how much from MAI-DxO’s multi-agent process.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Accuracy Gap Establishes
The reported scores suggest that AI systems can perform strongly on a particular type of diagnostic benchmark: difficult cases where information must be requested over several steps. That makes the study relevant to researchers and health systems considering whether AI could help organize information or support clinical reasoning. It does not show that MAI-DxO improves patient outcomes, reduces diagnostic errors in routine practice, or can safely replace a clinician.
The comparison is also not between equivalent care teams. The physician participants were primary care doctors and hospital generalists working alone, while the AI system used multiple virtual agents. The cases were selected as challenging records, and the source report says such cases may ordinarily be handled by specialists or physician teams. The results therefore cannot establish how the system would compare with specialists, collaborative clinical teams, or doctors using the full resources available in a hospital.
Cost figures deserve similar care. The benchmark assigned estimated prices to tests using billing codes and a 2023 U.S. hospital price list, plus a fixed physician-visit cost. The authors describe these as first-order approximations. They are not a demonstrated estimate of what a hospital would save by deploying the system, and they do not account for all costs or consequences of real-world care.
AI diagnostic tools for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
From Preprint to Science Paper
The paper, titled Sequential diagnosis with language models, appeared in Science, volume 394, pages 423–428, on October 22, 2026. The source report says it was received in April and accepted in August, and that all 15 listed authors were affiliated with Microsoft. The publication followed a June 2025 arXiv preprint that had not been peer reviewed and reported an earlier comparison using a different model and figures: 80% for the AI and 20% for physicians.
Microsoft’s accompanying blog post emphasized comparisons between the system and physicians, including the finding that MAI-DxO was more than twice as accurate as doctors using reference materials on complex cases. Those ratios describe Microsoft’s results on the study’s benchmarks. They should not be read as evidence of the same performance across all medical conditions or ordinary clinical encounters.
Some methodological details in the source report come from the earlier preprint, because the final Science paper is behind a paywall. Those details may differ in the published version. The preprint describes a model grading diagnoses against the recorded final diagnosis, with physicians re-grading a sample and showing good but imperfect agreement. It also says the gatekeeper could generate synthetic results when a requested test was absent from the record. Readers should treat these specific details as descriptions of the preprint unless confirmed in the final paper.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft CEO of AI Mustafa Suleyman, speaking to TechCrunch
As an affiliate, we earn on qualifying purchases.
Limits of the Case-Record Test
The study does not report how MAI-DxO performs when interacting with live patients, examining them, interpreting real-time imaging, or coordinating with clinicians. Its records were historical and text-based, so clinical safety and patient outcomes remain untested by this comparison. The source report also says the study’s cases contained no healthy patients and no benign explanations, preventing measurement of false alarms in those scenarios.
The size and nature of the physician comparison also matter. According to the source report’s account of the preprint, 21 U.S. and UK generalists took part, while the AI was assessed across all 304 complex cases and physicians were assessed on a smaller held-out sample. The source describes the Science paper’s open-book physician score, but says the preprint provides some of the more detailed information about participants and scoring. It is not clear from the supplied material whether every such detail remained unchanged in the final paper.
There is also no independent replication reported in the supplied material. Peer review evaluates a paper’s methods and reporting; it does not establish that results have been reproduced by independent teams or that performance will transfer to other hospitals, patient populations, and kinds of cases.
As an affiliate, we earn on qualifying purchases.
A Prospective Hospital Study
Microsoft’s next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. The source says consenting patients will interact with the system alongside their normal care. That design could test the system in a setting closer to clinical practice while patients continue to receive usual care.
The supplied material does not give a start date, enrollment target, study length, evaluation measures, or results timeline. Those details will matter when judging what the next study can establish—for example, whether it measures diagnostic accuracy, safety, effects on clinician decisions, patient outcomes, or time and cost. Until such evidence is available, the Science results remain evidence about performance on defined case-record benchmarks, not proof of clinical effectiveness.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Microsoft’s MAI-DxO study evaluate?
It evaluated how the system and physicians worked through historical medical case records by requesting additional information and tests before giving a diagnosis. The cases were presented in a simulator, not during care for live patients.
What accuracy did Microsoft report?
Microsoft reported 83.2% accuracy for MAI-DxO on 304 complex cases, compared with 36.1% for generalist physicians using textbooks and the internet and 22.3% without those resources. On 325 emergency department cases, the reported scores were 80.3% for the system and 46.8% for physicians.
Does the study show that MAI-DxO is better than specialists?
No. The comparison described in the source material involved generalist physicians working individually, not specialists or physician teams. The results do not establish how MAI-DxO performs against those groups.
Was the study conducted with patients?
No. It used historical, text-based cases in a simulated process. Microsoft’s reported next step is a prospective study at Beth Israel Deaconess Medical Center in which consenting patients would interact with the system alongside their normal care.
What remains unknown about the results?
The supplied material does not establish whether the system improves outcomes or safety in routine practice, how often it produces false alarms in a broader patient population, or whether independent researchers have reproduced the results. Details and timing for the planned hospital study were not provided.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
