UC Berkeley Researchers Break Top AI Agent Benchmarks

BERKELEY, CALIF. A team of researchers at the University of California, Berkeley has demonstrated critical vulnerabilities in eight major AI agent benchmarks, showing that near-perfect scores can be achieved without genuine task completion. The Center for Responsible, Decentralized Intelligence, led The research team found that a 10-line Python file added to conftest.py could resolve every instance on SWE-bench Verified. A fake curl wrapper achieved perfect scores on all 89 Terminal-Bench tasks without generating solution code. Navigating to a file:// URL allowed reading gold answers directly from task configurations, yielding approximately 100%% on all 812 WebArena tasks. The findings follow earlier instances of benchmark gaming. IQuest-Coder-V1 claimed 81.4%% on SWE-bench before researchers discovered that 24.4%% of its trajectories simply ran git log to copy answers from commit history. The corrected score dropped to 76.2%%. The researchers argue that current evaluation methods create perverse incentives. Models are optimized for leaderboard scores rather than genuine capability. The paper calls for benchmark designers to implement stronger security measures including isolated evaluation environments, cryptographic verification of task environments, and adversarial testing before publication. The research is available at github.com/moogician/trustworthy-env.