A Robust Multi-Validation Approach for Evaluating Machine Learning-Based Intrusion Detection Models

Intrusion Detection Systems (IDS) play a vital role in protecting modern networks from cyber threats by detecting abnormal or malicious traffic behaviors. Machine Learning (ML) techniques have been applied extensively to enhance automation, scalability, and detection accuracy. However, most ML-based IDS studies still rely on single validation schemes such as basic train-test split or Simple K-Fold Cross-Validation, which often produce biased estimates, overfitting, and poor generalization across datasets. This research presents a Multi-Validation Evaluation Framework designed to integrate six mutually supportive validation techniques: three single-validation methods (Hold-Out, Simple K-Fold, Stratified K-Fold), and three multi-validation methods (Repeated K-Fold, Bootstrapping, and Nested Cross-Validation), ensuring fair, consistent, and statistically reproducible assessment. The framework was validated on two benchmark datasets, NSL-KDD and UNSW-NB15, using five ML models: Random Forest, Extreme Gradient Boosting, Decision Tree, K-Nearest Neighbors, and Linear Support Vector Classifier. Model performance was evaluated using the Accuracy, Precision, Recall, F1-Score, ROC-AUC, and PR-AUC metrics. The outcomes are reported as mean ± standard deviation. The results show that Random Forest has the highest accuracy (99.56% and 94.69%) and ROC-AUC (>0.989) for all datasets. The multi-validation technique reduced metric variance by up to 40% while maintaining a mean accuracy steady, which shows that it is more stable and repeatable. Statistical tests (Wilcoxon, Friedman, and Nemenyi) showed significant disparities in performance (p < 0.001). The proposed method provides a robust, comprehensive, and scientifically valid framework to evaluate ML-based IDS models.