This paper proposes a mixed-effects machine-learning framework for modeling complex, nonlinear relations between predictors and continuous outcomes in multilevel cross-classified data.The proposed method, termed LMM-XGBoost, embeds gradient boosting (XGBoost) within a linear mixed model (LMM) to combine flexible modeling of nonlinear and interaction effects with random effects that model dependence.In addition, an iterative estimation procedure for LMM-XGBoost is developed, a group-aware permutation importance measure that respects multilevel dependence is proposed, and a combined-group cross-validation strategy for hyperparameter tuning, out-of-fold prediction, and importance estimation is developed for cross-classified designs.The simulation study shows that the proposed estimation method for LMM-XGBoost yields good parameter recovery under nonzero random-effect variances.In addition, relative to standard LMM and XGBoost, LMM-XGBoost achieves lower out-of-fold prediction error and more accurate recovery of variable importance.The study further shows that combined-group cross-validation and group-aware permutation importance yield less biased error estimates and substantially higher agreement with the true importance rankings than conventional permutation measures.An empirical application using the Add Health study illustrates how the proposed methods can identify important factors across multiple domains associated with adolescent depressive symptoms.
Cho et al. (Fri,) studied this question.