Self-Consistency Is a Weak Proxy for LLM Correctness, Large-Scale Study Shows

A large-scale study across 53 model runners and 265,000 samples on GPQA Diamond and AIME benchmarks found that agreement among model outputs -- whether from self-consistency or cross-model ensembles -- is a positive but weak predictor of correctness, with correlation coefficients ranging from 0.20 to 0.59. The relationship is regime-dependent: agreement helps most for mid-tier models and for compute allocation decisions, but becomes overconfident and no more accurate for frontier models. On GPQA, the most consistent frontier model agreed with itself on 77 percent of cases, yet 48 percent of those high-agreement answers were wrong. An exploratory check across three Claude model tiers showed the same pattern of confident errors recurring across providers. The authors conclude that self-consistency is a conditional proxy for correctness, not a standalone confidence score, and release de-identified per-run data and answer distributions.