CARGO study improves AI agent answer checks but misses procedural errors

Researchers testing a proposed method for evaluating AI agents found that it avoided false penalties when a correct answer used facts from a live case rather than identifiers from a reference case. In a 246-item diagnostic benchmark, standard reference-based judges penalized all 50 correct answers transferred to another case; the CARGO method penalized none while detecting 50 of 50 and 49 of 50 contradictory answers across two judge models. CARGO checks claims against the live case and withholds judgment when retrieval confidence is low. The preprint also found that its more lenient approach detected only 20% of procedural errors, a gap a subsequent fix did not close.