Microsoft's AI Diagnostic System Beats Physicians 83% to 36% on Text-Only NEJM Case Benchmark

A study published in Science on 22 October 2026 reports that an AI system built by Microsoft reached the correct diagnosis in 83.2% of 304 difficult case records from the New England Journal of Medicine, against 36.1% for generalist physicians working the same cases with reference materials. The paper, titled "Sequential diagnosis with language models," is authored by Harsha Nori, Dominic King, Mustafa Suleyman, Eric Horvitz and colleagues at Microsoft.

Testing diagnosis as a sequence, not a single guess

The authors built a benchmark called SDBench, which turns 304 New England Journal of Medicine clinicopathological conference cases, and 325 emergency-department admissions drawn from the MIMIC-IV-ED dataset at Beth Israel Deaconess Medical Center in Boston, into step-by-step encounters. Instead of handing a diagnostic agent the full case file, SDBench reveals an initial presentation and lets the agent ask questions and order tests one at a time, each carrying a simulated cost, before it commits to a final diagnosis. The system described in the paper, the Microsoft AI Diagnostic Orchestrator (MAI-DxO), coordinates several roles that resemble a panel of physicians with different specialties, revising its working diagnosis as answers and results arrive.

Paired with OpenAI's GPT-5, MAI-DxO scored 83.2% on the NEJM cases and 80.3% on the Beth Israel Deaconess emergency-admission cases, compared with 36.1% and 46.8% for the generalist physicians, according to the Science paper. The physicians were given access to reference materials such as textbooks and the internet but not to colleagues or specialists. A press release distributed by the American Association for the Advancement of Science adds that MAI-DxO's accuracy ranged from 79.6% to 84.5% depending on how tightly its test-ordering budget was constrained, with average simulated costs per case ranging from $1,303 to $8,945.

The two approaches also gathered information differently. According to the paper's results, MAI-DxO asked a mean of 25.5 questions before ordering its first test, versus a mean of 5.6 questions for the physicians in the study — a strategy the authors describe as extensive structured history-taking before ordering costly or invasive tests.

What the authors say the result does not show

The paper is explicit that its benchmark does not represent routine practice. Nori and colleagues write that the case distribution does not match that of real-world deployment, since NEJM clinicopathological conference cases are selected specifically for their diagnostic difficulty and teaching value, and the comparison used text-only case records rather than live patients, physical examinations, imaging or pathology specimens. The physicians in the study worked alone rather than as the teams or specialists who would typically be consulted on cases this complex.

A separate, independent evaluation of the paper published by Nabu Science, a platform that scores research papers on methodological rigour and reliability, rates the study "Limited, with caveats." It notes that the NEJM cases used to build SDBench are publicly available text that language models such as GPT-5 may have encountered during pretraining, and that the paper's "omission of possible pretraining exposure to public CPC cases leaves a central alternative explanation unaddressed." Nabu's scoring gives the paper 4.2 out of 5 for reporting and 3.9 out of 5 for contribution, but only 2.0 out of 5 for methodological rigour.

The same review notes that the reported gains from MAI-DxO's orchestration layer — an average of 4.4 percentage points on the emergency-department cases and 7.9 points on the NEJM cases — are measured against "minimally prompted" versions of the same underlying models rather than against stronger baselines, which it says limits how much of the improvement can be attributed to the orchestrator's design rather than to the choice of comparison point. Under a heading titled "Simulation constrains transfer to clinical practice," it writes that text-only interaction, a case pool selected for diagnostic difficulty, and cost estimates based on a single large U.S. health system's published pricing "leave substantial uncertainty about performance and cost trade-offs in routine care."

What it means for students and researchers

For anyone training to read AI-and-medicine papers critically, this is a useful case study in separating a benchmark result from a clinical claim. The authors themselves frame MAI-DxO as a research system, not a deployed tool: Microsoft states it is not approved for clinical use and still requires safety testing, clinical validation and regulatory review. Microsoft has published the underlying SDBench evaluation harness on GitHub, but there is no public app or API for the MAI-DxO orchestrator itself. Dominic King, one of the paper's authors, has made a related point in public comments about the artificiality of the test setup: patients do not arrive with a tidy case summary the way SDBench's inputs are structured.

The gap between the headline 83.2%-versus-36.1% comparison and the paper's own caveats about case selection, text-only presentation and unmeasured contamination risk is the part worth reading past the summary for. Researchers evaluating future sequential-diagnosis benchmarks will likely be watching for whether later studies report a pretraining-contamination check and compare orchestration layers against stronger single-model baselines, both of which this round of work leaves open.