Accurate Injury Severity Score (ISS) estimation is crucial for trauma care quality improvement but relies on labor-intensive manual review by trauma team and registrars. Large Language Models (LLMs) offer automation potential, yet their validation in trauma scoring is limited. This retrospective study from a Level I trauma center evaluated GPT-4, Grok-3, and Perplexity for automated ISS estimation. We used iterative prompt engineering to refine LLM and input the discharge diagnosis of trauma patients. Then we compared the LLMs-estimated ISS against human-assigned scores. A separate validation cohort was used to assess agreement via weighted Kappa statistics, Intraclass correlation coefficient (ICC) and Bland-Altman plots (LoA). The validation cohort had an ISS range of 1–41 (median: 9.0, IQR:12.0; mean ± SD: 11.71 ± 8.64). Categorical agreement was high in Grok-3 (weighted Kappa = 0.91), and GPT-4 (weighted Kappa = 0.88), but adequate in Perplexity (weighted Keppa = 0.54). All LLMs showed strong concordance with human-assigned ISS. ICCs were 0.97 (GPT-4, CI 95% = 0.94–0.98), 0.92 (Grok-3, CI 95% = 0.86–0.95), and 0.84 (Perplexity, CI 95% = 0.73–0.91), demonstrating near-expert-level consistency. Mean biases were small positive, with a LoA of ± 5.2 (GPT-4), ± 7.0 (Grok-3), and ± 9.7 (Perplexity). Performance decreased for severe injuries (ISS ≥ 16), with wider LoA and lower ICCs, likely due to less granular narrative documentation. LLMs demonstrate strong agreement with human-assigned ISS for automated estimation from clinical diagnoses. While complex severe injuries pose challenges, LLMs can significantly enhance efficiency and consistency in trauma registry workflows. Further validation and integration with richer data sources are recommended for clinical deployment.
Ouyang et al. (Mon,) studied this question.