How to create "humble" AI

An international group of scientists led by MIT has outlined a framework to design artificial intelligence systems that display greater humility when assisting with medical diagnoses and treatment recommendations. The researchers caution that current AI systems risk guiding doctors toward incorrect decisions because they may present overconfident but flawed outputs. Physicians have been shown in prior studies to defer to such systems even when their own judgment differs, the team noted. To address this, the framework incorporates modules that require AI models to assess their own certainty in predictions. One component, called the Epistemic Virtue Score, was developed by researchers at the University of Melbourne and serves as a self awareness check that tempers confidence based on the uncertainty in each clinical case. If the system detects that its confidence exceeds the supporting evidence, it can pause, flag the gap, and request additional tests, patient history, or specialist input, the scientists said. The goal is to position AI as a collaborative partner rather than an authoritative oracle, according to Leo Anthony Celi, a senior research scientist at MIT's Institute for Medical Engineering and Science. Celi is the senior author of the study, which appears in BMJ Health and Care Informatics. The lead author is Sebastián Andrés Cajas Ordoñez of MIT Critical Data. The work forms part of broader efforts by the global consortium to reduce biases in AI models trained on limited datasets and to involve clinicians, patients, and other stakeholders in system design. The research received funding from the Boston Korea Innovative Research Project through the Korea Health Industry Development Institute.