Abstract Number: Esoc2026a2380 a Novel LLM-Based Approach to Simulating Stroke Alerts for Neurology Residents

Abstract Background and aims While neurology residents play an integral role in acute stroke alerts at academic centers, few objective measures exist to track competency. We developed a virtual stroke alert simulator, on a customized large-language model (LLM) platform, to help educators assess performance. Methods Thirty participants from a single institution, including PGY1-4 neurology residents and attending physicians, each completed three LLM-based stroke alert simulations with distinct learning objectives. Eighty-seven conversation transcripts were independently graded by two blinded reviewers. We additionally evaluated model fidelity, medical accuracy and a novel metric assessing the LLM’s ability to control information release and provide details only when prompted. Results Performance was analyzed based on preset stroke competencies built into each case. In the first simulation, 60% of PGY1s correctly offered TNK after recognizing subtherapeutic warfarin compared to 75% of PGY2-4s and 83% of stroke attendings. In the second simulation, fundoscopy was performed by 0% of PGY1s, 30% of PGY2-4s, and 67% of stroke attendings for workup of CRAO. In the third case, 75% of PGY-1 residents correctly localized symptoms to spine, compared to 50% of PGY2-4s and 80% of stroke attendings. The model demonstrated 100% script adherence, 90% medical accuracy and 88.5% appropriate information control. Of 9 errors in medical accuracy, 8 occurred after simulations had ended and the model was given “free reign” to provide feedback. Conclusions LLM-based stroke alert simulations are feasible and demonstrate excellent fidelity. When case design is aligned with specific learning objectives, they can aid in assessment of stroke alert competency. Conflict of interest Dr. Harika Rayala would like to acknowledge support from the American Academy of Neurology for an Education Research Grant. Dr. Arlinda Deng: Nothing to disclose. Dr. Amna Sohail: Nothing to disclose. Dr. Alex Dalrymple is the UVA Neurology Residency Program Director and has no conflict of interest or funding relevant to medical education, technology or vascular neurology. Dr. Andrew Southerland has received personal compensation for serving as an Expert Witness for Plaintiffs and Defense Cases related to Stroke and Vascular Neurology as well as research support from NIH, American Heart Association, Abbvie Pharmaceuticals. Dr. Southerland has received intellectual property interests from a discovery or technology relating to health care.