Towards a Testbed for Improving Awareness of Computer-based Training Tasks

Trainers supervising computer-based training tasks require awareness of in-progress trainee activities, necessitating system support. It is our thesis that it is possible to design and implement the architecture and abstractions of an awareness-supporting layered testbed that can and does support new datasets, functionality, and evaluations in three important dimensions: task environments, algorithms, and awareness user interfaces.We develop new awareness algorithms that automatically group trainee solutions, enabling trainers to provide feedback at the group level. Not all groups require feedback, so we introduce misapplication detection, which identifies solutions that pass checks but reflect conceptual misunderstandings. Misapplication detection uses a keyword count embedding based on the insight that the distinction between expected and misapplied solutions lies in the combination and frequency of constructs. Both grouping and keyword-based detection require prior data, which is unavailable for new exercises, so we develop LLM-MAD, an approach that removes this dependency.We incorporate these inferences into both existing and newly designed awareness user interfaces. These interfaces rely on data collected as trainees work, which we obtain by logging actions performed within the computing environment used to carry out the task. We design a buffered, distributed architecture that leverages a new intermediate data language to promote component reuse. This architecture enables substantial reuse, supporting the integration of six task environments with limited redundant implementation.These integrated environments enable us to collect logs from over 100 assignments across nine course offerings and two teacher-training workshops. We use these logs to construct a labeled dataset, based on two exercises completed by over 150 students, for evaluating code grouping and misapplication detection algorithms. Based on this dataset, our keyword-based approach outperforms a baseline in both grouping and misapplication detection for in-progress and final solutions, while LLM-MAD achieves comparable performance without requiring prior data.We introduce a new metric that enables comparison of latency across alternative components. Using this metric, we show that three existing dashboards require different levels of computational resources to deliver action-inducing awareness within acceptable latency bounds.The collected data enabled new insights about the logged assignments. The system supported the research of approximately thirty students.