Preprint reports gains in detecting deception inside AI agents

Researchers report that monitors examining language models' internal activity outperformed monitors reading their actions on two coding benchmarks. Their preprint reports AUROC scores, which measure discrimination between cases, of 98.3% on SHADE-Arena and 98.0% on BashArena, compared with 97.7% and 97.6% for action-only text monitors. The approach combines information across model layers and tokens. Researchers also released the FIBS deception training dataset. In a separate evaluation, probes identified transcripts containing a model's true hidden goal with AUROC of up to 99.7%; these results concern benchmark evaluations.