The Quality Evaluation of Graduation Test Questions Based on Machine Learning Predictive Models for the Pass/Fail Outcome of the National Medical Licensing Examination

Background: In medical education, studies exploring the relationship between graduation exams, advancement exams, and the National Medical Licensing Examination (NMLE) are limited. Despite the active progress of machine learning (ML) in clinical AI (Artificial Intelligence), research on this topic in the context of exam that did in university and the NMLE is scarce. Methods: The study aimed to assess the relationship between answer results for obstetrics and gynecology questions in graduation exams and the NMLE pass/fail outcomes, the former of which were used as independent variables in ML and the latter of which the target variable as 1 or 0. We tried 13 types of ML models and selected the most appropriate ones. ML-predicted values equal to or above 0.5 were considered positive, while values below 0.5 were considered negative. Predicted values for student pass/fail were defined as AIpass Rate, and feature importance value for each question was defined as AIpass Index using Symbolic Regression. Results: The Random Forest model had a high AUC of 0.789 and accuracy of 0.813, while The XGBoost (eXtreme Gradient Boosting) model demonstrated a high AUC of 0.785 and precision of 0.855. Both models performed well in predicting NMLE outcomes. Regarding the student-focused AIpass Rate, all students with predicted values exceeding 0.793, which is 66.96% of the examinees, passed the NMLE; conversely, no student passed the exam with a score below 0.184. As for the question-focused AIpass Index, questions with high values, especially those reaching 0.999 and strongly correlating with NMLE pass/fail, suggest them as ‘good questions’. Conclusions: While predicting the outcomes of the NMLE in advance is considered challenging, understanding the characteristics of ‘good questions’ strongly correlated with the NMLE is crucial. By conducting a detailed analysis of their content, we suppose that the level of question creation by educators can be elevated, ultimately enhancing the quality of education for students.