Separate claim checks improve an AI vulnerability auditor in preprint test

Researchers report that checking tool claims in a separate model call improved an AI vulnerability auditor's accuracy by 12 points when it received false assurances. The preprint tested how models use evidence from static analysis, which examines code without running it. Across 16,572 audits, fabricated assurances that safety checks existed reduced detection of real vulnerabilities by up to 19 points relative to true notes. Fabricated alarms did not significantly reduce accuracy. Prompt instructions did not reliably curb this deference. Showing the auditor refuted claims produced the 12-point gain without reducing accuracy under true evidence, but a second model's smaller improvement was not statistically significant.