A hybrid machine learning and explainable AI framework for optimizing risk-based authentication

As online platforms continue to grow, the need for strong authentication mechanisms becomes increasingly important to protect sensitive information and networks. Risk-Based Authentication (RBA) is an adaptive approach that dynamically adjusts authentication decisions based on user behavior and contextual information, thereby improving both security and user experience. This study proposes a hybrid RBA framework that integrates machine learning ensemble techniques, fuzzy logic, clustering, and optimization to enhance account takeover detection and dynamic risk assessment. The ensemble classifier, combining Gradient Boosting, SVM, and XGBoost, predicts the probability of account compromise based on login behavior, device attributes, and network information. K-Means clustering is used to generate initial risk thresholds (low, medium, and high), which are further refined using a fuzzy logic system to map probabilities to risk levels. The L-BFGS-B optimization algorithm is employed to fine-tune fuzzy membership boundaries and improve threshold consistency. Experimental results demonstrate strong performance, achieving 97.77% accuracy, 99.41% precision, 98.04% recall, 98.72% F1-score, and an EER of 0.0303. On large-scale datasets ranging from 2M to 30M records, the proposed framework demonstrates consistent improvement in authentication decisions. For the 2M dataset, Allow Login actions increase from 349,432–349,923, while Deny Login actions decrease from 1,462–1,228, along with a slight reduction in additional authentication prompts. Furthermore, the use of Explainable AI techniques, particularly SHAP, enhances the transparency and interpretability of the model, supporting more informed decision-making. Overall, the proposed framework is accurate, adaptive, and suitable for real-world risk-based authentication applications.