Medical AI training reduces false lesion reports but misses critical external cases

Researchers report in a preprint that training MedScope-4B on images with lesions erased reduced its tendency to report findings absent from an image. On 507 internal cases across five imaging types, it claimed an erased lesion in 9.6% of cases, compared with 41-62% for the frontier models tested, without significant loss of accuracy on real lesions. The training combined erased lesions, control edits elsewhere and unedited lesion-free images to teach the model when to withhold a finding. External testing exposed a critical limitation: MedScope-4B detected 0 of 48 pneumothorax films in a fresh test. The authors conclude that the training changes reporting behavior without improving perception.