An Iterative Glmm–xgboost Algorithm With Group-Aware Conditional Permutation Importance for Explaining Multilevel Item Response Data

Identifying which individual-, cluster-, and item-level predictors explain binary responses requires balancing flexibility and statistical rigor.Generalized linear mixed models (GLMMs) explicitly partition fixed and random effects and capture multilevel structure, but they make it challenging to model nonlinear relationships and higher-order interactions.In contrast, tree ensembles such as XGBoost flexibly capture nonlinearities and interactions but typically ignore clustering and random effects.This study introduces an iterative GLMM-XGBoost algorithm that replaces the penalized weighted least squares step in the penalized IRLS (PIRLS) routine with a boosted-tree learner, while retaining Laplace-approximation-based updates of the random effects via their conditional modes.Weighted C-projection and global centering enforce orthogonality between the tree component and grouping factors, avoiding redundancy.The algorithm yields an approximate Newton-Fisher scoring update, preserves the ability to model random effects, and maintains the flexibility of XGBoost.In addition, a group-aware conditional permutation importance measure and its associated uncertainty measure are developed to identify predictor contributions under multilevel data.Results indicate that iterative GLMM-XGBoost improves predictive accuracy and importance ranking relative to standalone XGBoost under clustering and random effects.It also remains competitive with alternative methods across both smooth and tree-based fixed-effects specifications while accounting for random variation.