GitHub opens ReviewBench for evaluating AI code reviewers

GitHub released a research preview of ReviewBench, a public benchmark that lets developers evaluate AI code reviewers against a common dataset and scoring method. It contains 219 pull requests from 187 public repositories spanning 19 languages, with distributions informed by analysis of 103.9 million GitHub pull requests. The benchmark combines findings from human reviewers, analysis tools and multiple AI models, then uses Claude Sonnet 5 to judge their validity. GitHub says senior engineers independently relabeled the findings and agreed with its judgments 96.6% of the time. Developers can run their own agents and compare results by severity, category and tolerance for missed issues or false alarms. Leaderboard submissions require maintainer approval.