SANE preprint reports broader automated testing of AI judges

Researchers report that SANE, an automated framework for testing AI systems that score model outputs, produces 3.7 times as many validated test families as the Judge Reliability Harness at comparable end-to-end validity across five domains. The preprint reports that SANE detects scoring deficiencies in every domain, including professionalism and toxicity, where the comparison harness detects none. SANE derives tests from each scorer's requirements and checks whether generated cases support the expected scoring relationship. For faithfulness, its scores rank 11 judge models with a Spearman correlation of 0.852 against human-labeled FaithBench performance, compared with 0.100 for the comparison harness. That ranking result concerns faithfulness specifically.