Audit finds cheaper model can better judge text-to-SQL results

Researchers auditing a production system that converts questions into database queries found that its deployed GPT-4o-mini evaluator frequently disagreed with human reviewers. On a set enriched for disputed cases, it flagged 77.1% of queries humans judged faithful. A self-hosted Qwen3.6-27B evaluator achieved stronger agreement with human judgments and cost roughly 1/300 as much per call as Claude Opus 4.7, which scored similarly. The comparison between those two models covered 96 cases, limiting how firmly their agreement scores can be distinguished. The findings appear in a preprint.