AI can outperform physicians on a diagnostic test, but what happens when the patient forgets a symptom, the tests disagree, and the disease is still unfolding? A new study moves beyond the medical quiz toward the back-and-forth of diagnosis, with impressive results. It also reveals the distance between commanding medical knowledge and exercising clinical judgment.
“Clinical diagnosis is an iterative process in which physicians gather information from patient histories, physical examinations, and diagnostic tests to identify a medical condition. As new evidence emerges, they continually revise potential diagnoses and determine whether sufficient certainty exists to reach a conclusion or initiate treatment.”
Most studies reporting on AI’s ability to serve as a diagnostician, the first clinical step in care, have given AI systems full knowledge of the diagnostic query, “compressing diagnosis into one inference step.” This is not a true benchmark of clinical diagnostic competence, and while a new study reported in Science moves closer to real-world diagnosis, it remains a model and not a true comparison.
The key distinction in the new AI model was that it could work through an iterative information-gathering process, but it did so with a certainty that real-world practice lacks.
A Better Test, Still a Simulation
To reach a diagnosis, both the AI and human clinicians had to request additional history, examinations, and tests, and decide when they had enough information to diagnose. The cases were broken down into “bite-size” diagnostic nuggets from the New England Journal of Medicine’s weekly clinical pathology cases, which feature “interesting,” read rare or difficult diagnostic presentations, and from emergency department admissions — more the daily diagnostic fare for the human clinicians. [1]
The headline results were striking:
One might conclude that AI’s diagnostic accuracy was significantly better, though, as with much of healthcare, it came with additional costs [2]. When the AI was asked to be more parsimonious, it retained much of its diagnostic accuracy at lower cost. For example, in the NEJM cases, accuracy remained 80.9% at an estimated testing cost of $2,241.
While the researchers moved AI closer to realistic scenarios, before we turn over the diagnostic reins to a silicon chip, some details matter when interpreting their findings.
What the Headline Numbers Leave Out
-
The researchers chose those NEJM cases to present difficult diagnostic problems. AI systems are disproportionately exposed to interesting, rare cases, and their memory is always available. Physicians' exposure and memory are captured in their “experience,” which is rarely as broad and certainly not as easily searched.
-
The AI was designed to draw on multiple simulations to determine its next diagnostic move. The humans acted independently, without recourse to specialist consultation they might ordinarily use. As a result, the study compares a coordinated AI system with individual physicians. It is misleading to interpret the evidence as AI outperforming physicians working in teams and seeking guidance from other physicians, because that was not tested.
-
Perhaps the study’s most consequential limitation is that it removed much of the uncertainty inherent in clinical medicine. When information was unavailable, it generated plausible responses that offered no additional clues to the correct diagnosis. But real-world uncertainty is rarely so orderly. As Dr. House said, patients lie (or at least omit important details), tests conflict, and imaging findings are ambiguous, creating a messy clinical reality. The more predictable information-retrieval process the researchers designed eliminated much of the need to judge the reliability of evidence.
- The AI asked four times as many questions, but each question carries a cost: patient attention, clinician interpretation, clarification, and sometimes reassurance. Questions posed to a cooperative electronic intermediary are not equivalent to those posed to a frightened, confused, or exhausted patient.
Within the diagnostic “sandbox” the researchers created, AI was impressive, but that sandbox remains a highly simplified model of clinical reality. These are not estimates of the relative diagnostic accuracy of AI and physicians in ordinary practice. They are outcomes from a particular test, with particular patients, information rules, comparison groups, and definitions of correctness.
These limitations do not erase AI's performance advantage within the benchmark. They help explain what that advantage is and why access to an enormous body of medical knowledge isn't quite the same as clinical experience.
A Vast Memory, Available on Demand
AI models are trained on enormous collections of medical literature, case histories, and discussions spanning specialties and rare conditions. No individual clinician can acquire comparable breadth. However, the key term here is trained. These training sets are based on human evaluation and annotation through comments, statistical mentions, or peer review. The “weighting” of the contents of that knowledge is, at the least, biased by what information is used for training.
This latest AI improved upon its diagnostic accuracy by about 8% in the NEJM cases and 4.5% in the ED cases compared to a baseline prompt requiring no iterative diagnostic skills, not by “learning” new information, but by reorganizing that information in response to a new problem, comparing alternative explanations, and using the comparison to decide what information is needed next. That is a genuine form of sequential inference and clearly an improved modeling of a human diagnostician.
The critical advantage demonstrated by the AI is its broader access and speed of recall to encoded medical knowledge and now, the ability to apply structured reasoning to that knowledge with unusual consistency. However, this does not show that the AI has learned from the consequences of that encounter.
Experience Requires Consequences and Reassessment
A physician's experience is not simply a collection of remembered patient encounters. It incorporates the process of observing, interpreting, acting, experiencing consequences, and revising an internal understanding of disease — human evaluation and annotation, what we call learning. That ability is not merely retrieval. This study provides no evidence that the AI independently performs that kind of persistent revision.
From Knowing to Judging
Medical intelligence is not merely what we know or even what we remember from what we have seen. It is our ability to recognize relevant experiences, determine which new evidence matters, and decide when uncertainty justifies action rather than further investigation. What we term clinical experience develops through repeated cycles of perception, interpretation, prediction, intervention, and feedback. Over time, retrieving experience becomes faster and more contextual, and judgment determines which experiences matter and when to reconsider them.
AI has successfully modeled information gathering, interpretation, and diagnostic revision. It has not modeled intervention, its short- and long-term consequences, nor has it demonstrated reflection and persistent learning.
AI can bring a much broader range of diagnostic patterns to bear on each individual case. Its performance isn't constrained by fatigue, a particularly memorable recent patient, or one clinician's finite recall, although its outputs remain shaped by training data and prompting. It can also simulate multiple specialist perspectives, even when those specialists are not immediately available. In short, AI is currently much better at exploiting accumulated knowledge than at accumulating consequential experience of its own.
AI’s ability to bring an enormous store of medical knowledge to an unfamiliar case could be transformative, particularly for generalists confronting uncommon conditions. This study offers evidence that AI can gather information and revise a diagnosis within a structured simulation. Whether that performance translates into better care amid unreliable histories, conflicting findings, and evolving disease remains an open question.
Clinical experience develops through decisions and their consequences: what happened after treatment, which reassuring finding proved misleading, and when waiting became a mistake. The system evaluated here did not demonstrate that kind of continuing learning. The critical benchmark is whether AI can learn on its own.
[1] Even the emergency department cases represented patients sick enough to require admission, rather than the full range of complaints physicians routinely evaluate.
[2] Those costs are based on hospital-published charges, not actual costs.
Category
Chuck Dinerstein, MD, MBA
Director of Medicine
Charles Dinerstein, MD, MBA, is the Medical Director of the American Council on Science and Health. A vascular surgeon with more than 25 years of clinical experience, he brings a physician's understanding of medical decision-making, an appreciation for scientific uncertainty, and a broader perspective on healthcare economics and policy to his writing.
