Creating Exercises with Generative AI for Teaching Introductory Secure Programming: Are We There Yet?

Despite ongoing efforts to integrate security concepts into computer science curricula, many graduates still lack practical software security skills. Active learning strategies---such as drill-and-practice---offer a promising approach to bridging this educational gap. To implement these strategies effectively, educators must design and deliver hands-on exercises focusing specifically on secure programming. However, creating effective secure programming exercises is difficult, requiring substantial time and in-depth expertise. This paper examines the potential of generative AI to aid in creating drill-and-practice exercises for introductory secure programming settings. Specifically, we prompt several large language models (LLMs) to assist in generating exercises targeting three common software vulnerability classes, with tasks aligned to the advanced beginner stage of the Dreyfus skills acquisition model. We systematically evaluate the generated exercises for correctness and instructional viability. Our results show that, for some vulnerabilities, LLMs can produce technically sound and useful exercises for advanced beginners. While many generated exercises were near classroom-ready, minor fine-tuning is often necessary to ensure quality and pedagogical alignment. These findings suggest that effective exercise generation in secure programming is best achieved through a symbiosis between generative AI and human educators.