Accurate and efficient cardio-oncology outpatient clinic summarization with open-source large language models for non-english text

Abstract Background Clinical documentation poses a significant administrative burden in cardiology, but also in cardio-oncology, where comprehensive oncological history—including prior cancer diagnoses, treatments, and cardiotoxicity—are of vital importance. Large language models (LLMs) utilizing retrieval-augmented generation (RAG) offer a promising approach to partially automate this process. However, their effectiveness in most non-English texts remains largely unexplored. Purpose This study assesses the feasibility, accuracy, and efficiency of a locally deployed, open-source RAG-LLM system for automatically summarizing oncology and cardiotoxicity histories in Dutch medical records. Methods At our University Medical Center cardio-oncology outpatient clinic, 49 consecutive patients were included in this proof-of-concept study. All relevant oncology-related letters were extracted from the institutional data hub. Two investigators manually annotated prior cancer diagnoses, cancer treatments, and cardiotoxicity history to establish a "ground truth" dataset. A RAG pipeline was developed and deployed on the institutional high-performance computing cluster, using exclusively open-source components. The system employed LLaMA 3.1 8B, a high-performing midsize LLM, and enhanced response accuracy by retrieving only relevant text from the EHRs, thereby reducing hallucinations. The model automatically extracted the key variables from patient records. Three independent investigators evaluated the generated summaries by comparing them against the gold standard. Results A total of 49 patient records were analyzed, containing 374 documents. The most common prior cancer diagnoses were lymphoma (21/49, 42.9%) and leukemia (12/49, 24.5%). Nearly all patients received chemotherapy, with 24 (49.0%) undergoing radiotherapy. The system correctly identified 47/49 cancer diagnoses, 48/49 chemotherapy status, 45/49 radiotherapy status. For cardiotoxicity detection, the model correctly identified 9/12 positive cases and 35/37 negative cases, yielding a sensitivity of 75.0% and a specificity of 94.6%. The AUROC for cardiotoxicity discrimination was 0.849. The model demonstrated a strong understanding of Dutch medical terminology, with errors primarily stemming from retrieval limitations rather than incorrect reasoning. Retrieval inefficiencies also contributed to missed cancer treatment details. The system generated summaries in an average of 12.1 seconds per patient, compared to 3.5 minutes for manual extraction. Conclusion A locally deployed RAG-LLM system can accurately and efficiently summarize medical histories in a Dutch cardio-oncology setting. While the system demonstrated high accuracy in cancer diagnosis and treatment extraction, retrieval inefficiencies limited sensitivity for cardiotoxicity detection and comprehensive treatment identification.