Optimising Retraining Frequency for a Paediatric Emergency Department Admission Prediction Model: Development and Temporal Validation Using Real‐World Data

OBJECTIVE: To analyse temporal performance drift and optimal retraining frequency for an ensemble machine learning model to predict inpatient admission from paediatric emergency department (ED) triage data. METHODS: This study utilised 409,307 ED presentations from 1 July 2018 to 30 June 2024 at Perth Children's Hospital. An ensemble stacking model (XGBoost, TabNet, multi-layer perceptron and logistic regression base learners with a logistic regression meta-learner) incorporated structured triage features and tuned BioClinicalBERT-derived embeddings from free-text notes. The model ran prospectively through a 5-year rolling-window simulation, testing nine retraining cadences from weekly to triennial and a static model. Training, retraining and validation datasets were temporally separate and prior to the test set. Primary outcomes were discrimination via the area under the receiver operator characteristic (AUROC) and calibration as absolute mean daily bed error (AMDBE). RESULTS: Weekly retraining achieved a mean AUROC of 0.843 (SD 0.016) and AMDBE of 2.57 (SD 1.79) over the 5-year simulation. Fortnightly and monthly cadences were non-inferior (AMDBE 2.61 and 2.73), whereas longer intervals showed progressive calibration degradation (p < 0.001) and stable AUROC. Concept drift was most pronounced in the static model, with a mean AMDBE of 10.6 in 2024 compared to 1.79 for the weekly model. Notably, monthly retraining required only 25% of the weekly computational burden with non-inferior performance. CONCLUSION: Monthly, or more frequent, model retraining sustains discrimination and calibration for paediatric ED admission prediction. This effectively mitigated concept drift and enabled accurate simulated daily bed-demand forecasting, providing evidence to support the clinical testing of such modelling.