Abstract Background Cardiac magnetic resonance imaging (CMR) parameters are often stored in heterogeneous, unstructured clinical reports. Manual adjudication of these parameters is time-consuming and can require domain expertise. Recent open-source large language models (LLMs) have demonstrated impressive performance on language-task benchmarks. Moreover, they are cost-effective and readily adaptable to data privacy requirements and custom use-cases. To date, no prior study has systematically evaluated the performance of state-of-the-art (SOTA) open-source LLMs in extracting cardiac parameters from real-world CMR reports. Purpose To investigate and compare the ability of several SOTA open-source LLMs to automatically and accurately extract key cardiac parameters from unstructured CMR reports. Methods We retrospectively collected 1108 CMR reports from a single academic institution. Seven open-source LLMs varying in parameter size (2 to 9 billion) and pretraining corpus (general vs biomedical text) were evaluated, including Gemma 2-2B, Gemma 2-9B, Llama 3.2-3B, Llama 3.1-8B, Qwen 2.5-7B, BioMistral-7B, and Meditron 3-8B. We chose LLMs with fewer than 10B parameters to align with the computational capacity of commonly used GPUs in research settings. Each model was prompted to extract the following CMR parameters: cardiac output (CO), cardiac index (CI), left and right ventricular (LV/RV) ejection fraction (LVEF, RVEF), LV/RV end-systolic volume index (LVESVI, RVESVI), LV/RV stroke volume index (LVSVI, RVSVI), LV late gadolinium enhancement (LV LGE), LV LGE type (ischemic, non-ischemic, mixed), and T2 positivity. Results were pooled across three runs to account for LLM stochasticity. Concordance between model outputs and human expert-adjudicated values were computed. Results Gemma 2-9B achieved the highest concordance with human annotation on 6 out of the 11 CMR parameters, including CI (100%), CO (99%), LVEF (97%), RVEF (99%), LVSVI (97%), and RVSVI (99%), Figure 1A. Qwen 2.5-7B performed the best for LVESVI (98%), RVESVI (99%), and T2 positivity (95%). T2 positivity exhibited the highest average concordance across models (92.4%), while LV LGE type had the lowest (76.3%). Larger-parameter models (Gemma 2-9B, Llama 3.1-8B) consistently outperformed their smaller-parameter counterparts (Gemma 2-2B, Llama 3.2-3B), Figure 1A. Surprisingly, the two medical-domain models, BioMistral-7B and Meditron 3-8B, generally exhibited lower performance compared to the non-medical LLMs. Indeed, BioMistral-7B was the worst-performing LLM for all 11 CMR parameters, Figure 1B. This suggests that medical (domain-specific) pretraining may negatively affect adjudication performance. Conclusion Open-source LLMs show promise for automated and accurate extraction of cardiac parameters from unstructured CMR reports. Larger-parameter, general-pretrained LLMs—rather than LLMs trained on biomedical data—provide more accurate adjudications.Fig 1.LLM CMR adjudication performance
Wahi et al. (Sat,) studied this question.