Inference compute scaling lifts frontier LLM scores on hard benchmarks, arXiv study finds

A study posted to arXiv finds that frontier language model performance on difficult benchmarks improves substantially when models are given more inference-time compute, and that fixed-budget evaluations increasingly understate the capabilities of newer models. Researchers evaluated up to 12 frontier models on seven benchmarks spanning software engineering, mathematics, medicine, and cybersecurity using three inference-scaling interventions: larger token budgets, context compaction, and repeated submission attempts guided by the model or minimal correctness feedback. They found that larger token budgets consistently raised scores across domains including FrontierMath, Humanity's Last Exam, TerminalBench, and cybersecurity tasks. The paper argues that benchmark scores are protocol-dependent: fixed-budget evaluations can understate capability as models advance because newer models reach higher performance at large compute budgets where they unlock and solve harder tasks more reliably. Different benchmarks also respond differently to scaling methods, repeated submission broadly helps, while the value of larger token budgets, external feedback, and parallel attempts varies by benchmark. The authors recommend that evaluations report capability as a function of inference-time compute, specify protocol choices explicitly, and compare model generations over a large shared compute range at matched budgets, especially for safety- or policy-relevant assessments.