How Novices Use LLM-based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment

As Large Language Models (LLMs) gain in popularity, it is important to understand how novice programmers use them and the effect they have on learning to code. We present the results of a thematic analysis on a data set from 33 learners, aged 10-17, as they independently learned Python by working on 45 code-authoring tasks with access to an AI Code Generator based on OpenAI Codex. We explore several important questions related to how learners used LLM-based AI code generators, and provide an analysis of the properties of the written prompts and the resulting AI generated code. Specifically, we explore (A) the context in which learners use Codex, (B) what learners are asking from Codex in terms of syntax and logic, (C) properties of prompts written by learners in terms of relation to task description, language, clarity, and prompt crafting patterns, (D) properties of the AI-generated code in terms of correctness, complexity, and accuracy, and (E) how learners utilize AI-generated code in terms of placement, verification, and manual modifications. Furthermore, our analysis reveals four distinct coding approaches when writing code with an AI code generator: AI Single Prompt, where learners prompted Codex once to generate the entire solution to a task; AI Step-by-Step, where learners divided the problem into parts and used Codex to generate each part; Hybrid, where learners wrote some of the code themselves and used Codex to generate others; and Manual coding, where learners wrote the code themselves. Our findings reveal consistently positive trends between learners’ utilization of the Hybrid coding approach and their post-test evaluation scores, while showing consistent negative trends between the AI Single Prompt and the post-test evaluation scores. Furthermore, we offer insights into novice learners’ use of AI code generators in a self-paced learning environment, highlighting signs of over-reliance, self-regulation, and opportunities for enhancing AI-assisted learning tools.