Abstract Background: Surveillance activities are emerging as exemplar use cases for large language models (LLMs) in health care. The aim of this study was to evaluate the potential for LLMs to support the expansion of surveillance activities to include cardiovascular implantable electronic device (CIED) procedures. Methods: A validated machine learning-based infection flagging tool was applied to a cohort of VA CIED procedures from 7/1/2021 to 9/30/2023; cases with ≥10% probability of CIED infection underwent manual review. Then, a weighted random sample of 50 infected and 50 uninfected cases was reviewed with generative artificial intelligence (GenAI) assistance. GenAI prompts were iteratively refined to extract and classify all components of infection-related variables from clinical notes. Data extracted by GenAI were compared with manual chart reviews to assess infection status and extraction consistency. Results: Among 12,927 CIED procedures, 334 (2.58%) had ≥10% probability of CIED infection. Among 100 sampled cases, 50 of 50 uninfected cases were correctly categorized. Among 50 infection cases, GenAI identified all CIED infections, but the timing of events and the attribution to a preceding procedure were incorrect in 7 of 50 cases. The overall specificity of the GenAI-assisted process was 100% and the sensitivity for accurately classifying timing and attribution of CIED infection events was 82%. Errors in timing improved with iterative prompt updates. Manual chart reviews averaged 25 minutes per chart; the GenAI-assisted process averaged 5–7 minutes per chart. Conclusions: LLMs can help streamline the review process for healthcare-associated infection surveillance, but manual adjudication of output is needed to ensure the correct timeline of events and attribution.
Basnet et al. (Fri,) studied this question.