The Use of Large Language Models Tuned with Socratic Methods on the Impact of Medical Students' Learning: A Randomised Controlled Trial (Preprint)

BACKGROUND Large Language Models (LLM) are AI models that can generate conversational content based on a trained specified source of information (corpus). OBJECTIVE The aim is to use these corpus-trained LLMs to limit the content offered by LLM, then using prompt engineering to teach using Socratic methods. METHODS Two chatbots were created and deployed, powered by OpenAI’s GPT-3.5 model, with a medical-school textbook corpus. The first chatbot generates a brief summary and open-ended question. The second chatbot generates a case vignette from its pre-trained clinical cases, prompting users for a diagnosis. Both chatbots reply to the user’s response, commenting on the accuracy and asks further questions to encourage critical thinking. A randomised controlled trial was conducted on two groups comprising third year medical students. One group used both chatbots for 10 minutes while the other read the medical textbook. A 15-question test was administered to both groups before and after the intervention. RESULTS Forty students participated in the study. The average of the group before and after reading the textbook (n=20) are 3.9 +/- 1.0 and 7.6 +/- 1.5 respectively (p CONCLUSIONS Medical students’ learning showed a better performance using a LLM based chatbot compared to self-reading of medical information assessed using a standardised test. More studies are required to determine if LLM-based pedagogical methods are superior to standard education.