Representation Learning of Problem-solving Behavior in Virtual Reality Manufacturing Environment Using Long Short-term Memory Networks

Abstract The number of manufacturing jobs in the US has been consistently increasing, driven by a rapidly evolving industrial landscape and the implementation of a new strategic plan. At the same time, concerns have emerged about the problem-solving abilities of engineering students, who represent the future workforce. This highlights the need for systematic evaluation and deeper insight into how these students approach problem-solving. In this paper, we introduce a virtual reality (VR)-based manufacturing environment combined with a data-driven analytical workflow to evaluate engineering students' problem-solving performance. Within the VR system, students complete assembly tasks to build car toys that meet specific design criteria. During the process, we capture real-time eye-tracking data, reflecting the spatial and temporal dynamics of their visual attention. We extract latent features from this data via a long short-term memory (LSTM)-based supervised representation learning for problem-solving performance evaluation. Our approach outperforms the traditional performance metrics-based evaluation by capturing the nonlinear dynamics from the in situ production process. Experimental results show that the learned feature representations provide significantly clearer distinctions in categorizing the students' problem-solving performance compared to the traditional performance metrics. The proposed evaluation framework holds broader potential for improving problem-solving assessments in various manufacturing systems and workforce training programs.