In a striking development that could reshape the future of healthcare, a new study has found that OpenAI's advanced reasoning model significantly outperforms experienced human doctors when it comes to diagnosing rare diseases in emergency settings.
Led by researchers from Harvard Medical School and published in the journal Science, the study tested OpenAI's "o1 preview" model on 76 real-life emergency room cases from a Boston hospital. The researchers asked the model to diagnose a patient's condition and create a suitable testing plan. They then compared its clinical reasoning skills to those of an expert human doctor.
The AI was evaluated at three critical stages: initial triage, first doctor consultation, and final admission decisions, and compared against seasoned physicians.
ALSO READ: The Dog That Can Save Your Life: Bengaluru's Dognosis Trains Canines To Detect Cancer From Breath
Two independent doctors, who didn't know whether the assessments came from a human or the machine, evaluated the AI's performance as equal to or better than that of experts at every stage. The model performed particularly well during the initial triage phase, when there was the least information available.
The AI performed exceptionally well in identifying rare and complex medical conditions. It performed strongly on the challenging diagnostic cases from Massachusetts General Hospital, which have served as a benchmark for computer-based diagnostics since 1959.
In terms of clinical decision-making or 'management reasoning' like suggesting antibiotics or carrying out sensitive end-of-life discussions, the AI clearly outperformed both previous AI systems and human doctors with the help of tools like the latest Google Search.
After achieving these impressive results with the OpenAI model, the study's authors emphasized that their findings do not imply AI should replace doctors, a claim that some companies might be quick to make. Senior co-author Arjun Manrai said the study relied solely on text-based inputs, an area where language models perform very well.
However, he said that real-world physicians perform different practises. "They have to listen to the patient, they have to review chest X-ray radiographs, imaging studies, and they have to use lots and lots of other types of data—physiological signals, EKGs, ECGs—in everyday clinical decision making."
ALSO READ: Chronic Burden: Hypertension, Diabetes, Cholesterol Drugs Dominate 40% Of Indian Pharma Landscape
The study's senior co-author, Adam Rodman, an assistant professor at Harvard Medical School, highlighted two promising areas where AI could be particularly valuable: assisting with triage by reviewing electronic health records to catch missed diagnoses, and offering a reliable second opinion. According to a 2025 Elsevier study, one in five clinicians was already turning to AI for second opinions.
Essential Business Intelligence, Continuous LIVE TV, Sharp Market Insights, Practical Personal Finance Advice and Latest Stories — On NDTV Profit.
