Claude leads in small study of research approval checks
A University of Bristol-led study co-authored by the Health Research Authority (HRA) found substantial differences in how accurately AI models checked information intended for research participants. The peer-reviewed proof-of-concept study, published in Nature Scientific Reports, tested Claude 3.5 Sonnet, ChatGPT and Microsoft Copilot against specified quality standards. Conducted in 2024, it used 6 historical research applications.
Claude answered 87.9% of checks fully correctly, compared with 66.5% for ChatGPT and 62.8% for Copilot. It was also fastest, averaging 7.17 minutes per application, compared with 13.58 minutes for ChatGPT and 23.25 minutes for Copilot, and answered every question. HRA researchers with relevant expertise evaluated the responses without knowing which model produced them.
Failures of logical reasoning were the most common errors across the models. Claude recognised and corrected errors when challenged, while ChatGPT and Copilot sometimes defended incorrect answers by citing nonexistent page numbers. Routine checks can take more than a day, so AI assistance could free staff to focus on complex regulatory issues and improve assessment consistency.
The research covered routine checks before committee or staff review, not ethical decision-making. Its small scale and the possibility that model capabilities have changed since testing limit the findings. The HRA is continuing to evaluate safe and responsible uses of AI in research approvals.