MIT study: medical AI explainability helps non-experts most when they over-trust models

A Nature Medicine study led with MIT researchers finds AI assistance generally improved accuracy for non-experts and clinicians diagnosing skin diseases, but explainability methods affected groups differently based on expertise. Non-experts gained accuracy largely through deference to the AI. They trusted large language model explanations whether right or wrong and found vague or generic explanations more convincing. Clinicians resisted incorrect AI assistance and performed best when given only a model prediction with no accompanying explanation. Researchers tested approaches including confidence alone, similar-image reinforcement, heat maps, and plain-language LLM rationales. A fairness-constrained model designed to combat bias against darker skin tones significantly improved accuracy and reduced diagnostic disparities by skin tone. Presenting an explanation before users formed their own diagnosis increased deference. The work appears in Nature Medicine and was funded in part by the National Science Foundation, Schmidt Sciences, the National Bureau of Economic Research, and Columbia University.