Preprint tests uncertainty estimates for closed language models

Researchers introduced Pinocchio, a tool that estimates whether a language model's answer is correct without access to the model's internal data. In a preprint, they report that a version trained on responses from seven models scored 0.862 on a measure of how well it distinguished correct from incorrect answers held out from those models. The tool needs one processing pass and also transferred to thirteen previously unseen models across eight organizations. The researchers released code, but the reported score comes from models represented in its training data.