Divide and recombine approaches for fitting logistic regression to large-scale health surveillance data: application to diabetes risk prediction in BRFSS

The global rise in diabetes mellitus poses a major public health concern, highlighting the urgent need for effective risk prediction models to support early identification and prevention strategies. Yet, applying conventional statistical techniques to large-scale health surveillance data remains challenging due to memory limitations and high computational demands. One promising alternative, the divide and recombine (D&R) approach partitions the data into smaller subsets, fits logistic models independently within each, and combines the estimates to approximate full-sample maximum likelihood results. While the D&R methodology and its theoretical properties are well established in the statistical literature, such validation on national-scale health surveillance data remains limited. This study provides a comprehensive empirical validation of the D&R strategy for fitting logistic regression models efficiently on massive datasets, demonstrating its computational scalability, statistical robustness, and reproducibility in a real-world public health setting. We utilize the Behavioral Risk Factor Surveillance System (BRFSS) data from 2014 to 2024, encompassing over 2.4 million observations and 16 demographic, behavioral, clinical, and socioeconomic predictors. Monte Carlo simulations using 5 million synthetic observations and 1,000 replications confirm that the D&R method matches the statistical efficiency of centralized estimation (relative efficiency > 99.8%) while reducing computation time by more than 52% and memory usage by 77–89%. Application to BRFSS data further demonstrates the framework’s practical scalability, successfully recovering well-established diabetes risk factors including age, body mass index, general health status, alcohol use, cardiovascular disease history, and smoking, consistent with the epidemiological literature. These findings confirm that the D&R framework enables large-scale chronic disease modeling on standard computing infrastructure without reliance on high-performance systems, with direct implications for population health monitoring and preventive care resource allocation.