A24-08 In-hospital Cardiac Arrest - Using a Large Language Model to Identify Cases

Abstract Rationale In-hospital cardiac arrest (IHCA) is a catastrophic event associated with a high risk of mortality. IHCA research is limited by the low sensitivity of administrative data and the absence of a structured electronic health records (EHR) field to reliably capture these events. Large language models (LLMs) offer a unique opportunity to identify and extract salient information about cardiac arrest from clinical texts. Methods We used EHR data from an 11-hospital health system formatted using the Common Longitudinal ICU data Format (CLIF). Our convenience sample included 311,000 random notes from adult hospital encounters between 2020 and 2022. We used a zero-shot Llama 8b model to classify each note for the occurrence of an IHCA and to extract an event timestamp and a 200-character snippet as evidence. The prompt was, “You are a clinical NLP assistant. You read raw clinical text and decide if a definite in-hospital cardiac arrest event happened (not a risk, concern or rule-out). Only consider the actual text and extract the best timestamp for the event if present.” We also identified IHCA using International Classification of Diseases, Tenth Revision (ICD-10) (not present on admission) and Common Procedural Terminology (CPT) codes. We analyzed the 90-day mortality stratified by identification method (either ICD-10 or notes-only based). Results There were a total of 311,000 notes representing 26,207 unique patients. The median age was 49 IQR 33-67, 57.6% were female, and 74.9% were non-Hispanic White. There were 264 (1.0%) IHCA events identified using ICD-10/CPT codes and an additional 259 (∼1%) identified using only LLM-model. A total of 1985 notes were marked as having an IHCA event, representing 462 patients. Using 150 manually reviewed notes, the positive predictive value (PPV) was 26/100 (26%) in the patients without corresponding ICD-10 codes and 30/50 (60%) in the patients with corresponding ICD-10 codes. The model captured instances of IHCA that did not meet the clinically accepted context including pulselessness during death examination or cardiac surgery requiring cessation of cardiac activity. Conclusions Despite cardiac arrest being a highly documented event with unique and distinct terminologies that would otherwise not be used routinely (i.e., cardiac arrest, CPR), an open-source foundation model had low PPV in identifying IHCA through clinical notes. Future methods should focus on fine-tuning LLM models using manually curated and enriched notes from ICD-10/CPT identified patients. This abstract is funded by: None