Abstract Purpose This study evaluates Large Language Models as automated analyzers for detecting students’ erroneous mental models in programming education, addressing the scalability barrier of labor-intensive manual analysis. Building upon preliminary GPT-4 findings, this investigation extends to GPT-5, assessing generational improvements in reliability, error patterns, and computational feasibility. Methods GPT-4 and GPT-5 analyzed 1, 500 student responses to 24 C++ programming questions, yielding 5, 300 binary judgments for 27 hypothesized mental models compared against expert faculty evaluations. Inter-rater agreement was quantified through percentage agreement and Cohen’s Kappa, with confusion matrix analysis revealing systematic error patterns. Results GPT-5 achieved 82. 7% raw agreement with faculty (vs. GPT-4’s 66. 9%) and Cohen’s Kappa of = 0. 38 (vs. 0. 18), though below the 0. 60-−0. 80 inter-human threshold. Both models exhibited false positive bias, with GPT-5’s reduced rate (11. 6% vs. 23. 1%) driving improvement. GPT-5 demonstrated 7. 6-fold speed improvement and 34% cost reduction (12. 51 vs. 19. 06). Conclusion Current LLMs achieve production-ready status within hybrid human-AI frameworks. GPT-5’s fair-to-moderate reliability combined with cost-performance improvements (50 per 100-student cohort) establishes feasibility at educational scales. However, reliability insufficient for autonomous deployment supports hybrid configurations wherein LLMs serve as first-pass analyzers subject to systematic faculty review.
Gallego-Durán et al. (Tue,) studied this question.