28: Pediatric Risk of Illness Mortality Evaluation (Prime) Performance at Quaternary Children’s Hospital

Introduction: There is an ongoing need for generalizable risk adjustment models within intensive care and a growing need to update pediatric models, which were developed using data from more than 10 years ago. Curated real world data sources, such as the electronic health record (EHR), provide the ability to incorporate more information and automate calculation. We developed the Pediatric RIsk of Mortality Evaluation (PRIME) and evaluated its ability to predict mortality in critically ill children. Methods: This is a retrospective cohort study of admissions to a quaternary PICU from Oct. 2015-Dec. 2021. EHR data were curated from the first 24 hours of PICU admission. Patients were assigned randomly to training (70%) or test (30%) datasets. A ridge logistic regression model was trained on a hospital mortality outcome, with tuning performed using 10-fold cross-validation. Extreme gradient boosted and random forest models were trained for comparison. Model performance was assessed by examining area under the receiver operating curve (AUROC), area under the precision recall curve (AUPRC), standardized mortality ratios (SMR), calibration plots, Brier scores, and the Hosmer-Lemeshow test; performance was compared using the DeLong test. Data are presented with counts (%), medians (interquartile range), and 95% confidence intervals. Results: There were 15241 encounters with 286 (1.9%) deaths. There were 6738 (44.2%) females and median age was 6 (1-13), with no significant differences between the training and test datasets. There were 7964 (52.2%) admissions from the ED, 2786 (18.2%) from other hospitals, 2260 (14.8%) from acute care, 1918 (12.5%) post-operatively, and 313 (2.1%) from other ICUs. Primary admission reasons were respiratory (5365, 35.2%) and injury/toxic ingestion (2093, 13.8%). PRIME demonstrated excellent discrimination with an AUROC of 0.93 0.91-0.96 and an AUPRC of 0.46 0.35-0.57. Visualized calibration was good, the Hosmer-Lemeshow statistic had P=0.47, the SMR was 1.0, and Brier score was 0.013. PRIME significantly outperformed the electronic Global Open Source Severity of Illness Score (eGOSSIS; P=0.01). Conclusions: PRIME is a modern, high-performing mortality prediction algorithm. It will be refined using a multicenter international cohort and validated on holdout data from after 2021.