Abstract This study investigates the performance and fairness of predictive models designed to predict data science programming performance using fine-grained log data from a sample of students solving Data Science tasks in DaTu . Our analysis reveals that Logistic Regression outperformed other models in terms of accuracy. Through Variable Importance analysis, we found that the 'paste answer' variable emerged as a critical predictor for Logistic Regression and KNeighbors classifier models, whereas multiple variables contributed equally to the GaussianNB model's predictive power. We evaluated fairness, focusing on the variable 'chatbot access' (with or without access to chatbot), finding evidence of bias. The models demonstrated lower score distributions for the unprivileged group (no-AI access). To mitigate this bias, we employed various techniques: Resampling, Reweighting, and ROC pivot. Of these, only Reweighting significantly improved fairness metrics. As an alternative approach, we considered the ceteris paribus cutoff method to minimize parity loss. In conclusion, this study emphasizes the significance of evaluating fairness in educational predictive modeling. This work offers a nuanced and data-driven approach to assessing students' data science programming skills.
Doleck et al. (Mon,) studied this question.