METR updates developer productivity experiment design for AI tools study

METR said it is updating the design of its developer productivity experiment because of issues with its latest study. The organization started a new experiment in August 2025 after an earlier paper found AI tools slowed task completion by 20 percent among experienced open source developers. Participant feedback showed many developers now decline to join studies that require working without AI. The pay rate was also lowered from 150 dollars per hour to 50 dollars per hour, contributing to selection effects that likely bias results downward. METR reported raw data from the new study indicated a speedup of 18 percent for some original participants and 4 percent for new ones. It added that the true effect could be larger among those who opted out of the experiment. The organization plans to address these problems with changes including more intensive experiments and other research methods. These include observational data from AI use in software development and fixed-task experiments.