What question did this study set out to answer?

The aim is to assess automated LLM methods for identifying acute NAION in patient records, improving accuracy and efficiency over traditional methods.

June 4, 2026Ophthalmology Science0 citationsOpen Access

Automated Review of Patient Records: Privacy-Preserving LLMs for identifying incident NAION at Scale

TNTuyet Thao Nguyen KLKelvin Zhenghao Li PCPareena Chaitanuwong

Key Points

The aim is to assess automated LLM methods for identifying acute NAION in patient records, improving accuracy and efficiency over traditional methods.
Retrospective cross-sectional design involving 165 patients with ICD-10 codes for ION.
Utilized five LLM models to classify acute NAION based on unstructured ophthalmology records.
Compared LLM classifications to diagnoses made by expert neuro-ophthalmologists.
LLM approaches demonstrated a positive predictive value (PPV) for identifying acute NAION cases compared to expert review.
Results showed improved sensitivity, specificity, and accuracy over traditional ICD-10 coding methods.
The study evaluated multiple LLM approaches for efficacy in case identification against established expert standards.

Abstract

Purpose: Retrospective identification of acute non-arteritic anterior ischemic optic neuropathy (NAION) cases is critical for research on risk factors.However, reliance on ICD-10 coding for case identification has limited accuracy, and manual review of longitudinal electronic health records (L-EHR) is time-intensive.The purpose of this study is to evaluate automated methods for retrospective identification of acute NAION cases using large language models (LLMs) that preserve patient privacy. Design: Retrospective cross-sectional studySubjects: 165 patients with ≥1 ICD-10 code for ION (H47.01*) in the electronic health record at an academic medical center Methods: Five locally deployed LLM models (Mistral Small 3.1, Magistral Small, Gemma3, MedGemma, GPT-OSS 20B) were used to implement four approaches for acute NAION diagnostic classification using unstructured ophthalmology records (basic prompting, retrievalaugmented generation (RAG), two-step agentic workflow, and three-step agentic workflow).10% of subjects were used for prompt refinement.LLM/approach diagnostic classifications were compared against expert neuro-ophthalmologist diagnosis based on chart review.Main outcome measures: positive predictive value (PPV) of LLM approaches for acute NAION case identification with expert chart review diagnosis serving as gold standard.Secondary outcomes included negative predictive value, sensitivity, specificity, accuracy, F1 score and distribution of LLM/approach classifications.

Ask AI

Helpful

Bookmark

View Full Paper