From heartbeats to actions: Multimodal learning analytics of cognitive and behavior engagement in real classrooms

In the classroom, there may be a mismatch between students’ external behavior and their internal cognitive states. For example, students may exhibit attentive behavior but not be engaged in deep cognitive processing. This study combined time-on-task and heart rate variability (HRV) data to explore the dynamic relationship between student behavior and cognition. By comparing Video Learning Classrooms (VLC) with Traditional Learning Classrooms (TLC), the influence of different teaching settings on students’ learning states was explored. 45 students (paired sample) from German vocational schools and high schools. Time-on-task in the classroom was coded (in 10-s intervals) as a measure of students' dynamic behavior. The HRV indicator RMSSD (root mean square of successive differences) measured students’ dynamic cognition. Kruskal-Wallis tests revealed significant differences in HRV between the two classroom settings for all time-on-task categories (p < 0.001). Spearman's correlation analysis showed a significant negative correlation between time-on-task and HRV (VLC: ρ = −0.1621; TLC: ρ = −0.2184). In the VLC, students had more cognitive load when performing learning tasks, but the cognitive fluctuations were more stable. The TLC, on the other hand, showed greater cognitive fluctuations. Combining behavioral and physiological data using the MMLA can more accurately capture students' learning states. The study found that the impact of different instructional settings on students' learning states differs. Video tutorials help to stabilize students' cognitive engagement, while traditional classrooms place higher demands on students’ self-regulation abilities. • Multimodal analysis in an authentic classroom. • Heart rate variability mapping dynamic cognitive changes in different lesson phases. • Patterns of cognitive and behavior engagement in video versus traditional instruction.