Does a machine learning-based clinical insight bot accurately and efficiently identify cardiopulmonary disease cases for preoperative risk stratification from unstructured clinical notes?
An NLP-based machine learning model can accurately and efficiently identify preoperative cardiovascular risk signals from unstructured clinical notes with high precision and low false positive rates.
BACKGROUND: Preoperative chart review is time-consuming and prone to errors, particularly for cardiopulmonary conditions that impact anesthetic planning. We developed a guideline-aligned “clinical insight bot” that mines free-text documentation to surface perioperative cardiovascular risk signals relevant to the 2024 Mult Society perioperative guideline for noncardiac surgery. METHODS: We analyzed 1000 de-identified medical cases from the PhysioNet MIMIC database. Medical terminology was extracted using regex-based NLP and categorized into 13 clinical specialties. Text features were encoded using TF-IDF vectorization and 1536-dimensional semantic embeddings stored in a PostgreSQL vector database (pgvector). Four machine learning models—Logistic Regression, Random Forest, Support Vector Machine (SVM), and Naive Bayes—were trained with stratified fivefold cross-validation to classify cases as “cardiopulmonary-only” versus “mixed/other.” Performance was evaluated using accuracy, precision, recall, and F1 score, with statistical comparison via McNemar’s test and bootstrap confidence intervals. RESULTS: In a held-out test set of 200 notes (28 positive; 172 negatives; ~14% prevalence), a linear support vector machine achieved the best overall balance (F1 ≈ 0.71), with high precision (positive predictive value 0.94) and very low false positive rate (FPR) (1/172 ≈ 0.6%). False negatives were the dominant residual error class. The pipeline processed documents near-instantaneously and, when scaled to 1000 notes, replaced on the order of tens of clinician review hours (≈100× efficiency gain) while maintaining performance across common preoperative document types. CONCLUSIONS: A lightweight, guideline-aligned insight bot can transform unstructured preoperative notes into concise, stepwise prompts that flag cardiovascular risk signals before the day of surgery. High precision with a very low FPR supports safe integration with anesthesiology workflows by minimizing paging noise, whereas time savings create operational and financial value. Future work should emphasize multicenter validation, structured data fusion (including labs, imaging, and vitals) to improve sensitivity, and prospective evaluation of downstream clinical and operational outcomes.
Aggarwal et al. (Wed,) studied this question.