Study finds AI exam answers outperform many law students

Researchers testing nine AI models on University of Wollongong law exams found that their criminal law answers averaged 76.3%, above the scores of 82.5% of students. In tort law, the AI answers averaged 66%, above 61% of students. The results mark a change from the researchers' 2023 criminal law test, when AI answers averaged 52.5% and scored around the 22nd percentile. The models produced 18 papers across two subjects, and some answers had weak citations or fabricated authorities. The authors say the results raise concerns about unsupervised assessment, while the small, single-university test does not establish that the models are reliable legal experts.