Sensor-based modeling of problem-solving in virtual reality manufacturing systems

Problem-solving is the process to achieve a goal when the solution path is uncertain. Recently, technological advancements have changed problems’ characteristics and their solutions in engineering fields. Strong problem-solving skills are essential to allow engineers to assess new problems and quickly implement solutions. Engineering problem-solving skills are first educated in schools and usually evaluated through written exams. However, high grades in exams do not represent sufficient problem-solving skills in real-world engineering problems. Decision making with insufficient problem-solving skills in real world may result in costly consequences. Therefore, it is imperative to evaluate and reinforce problem-solving skills of engineering students in real-world problems. With the rapid technological advancements, availability of virtual reality (VR) and eye-tracking facilitates the study of engineering problem-solving. The immersive environment created by VR enables students to better understand and solve real-world engineering problems. On the other hand, eye-tracking allows for studying fundamental cognitive processes during information processing. It is critical to integrate VR simulation with data-driven modeling of eye movements to evaluate and enhance engineering problem-solving skills. In this paper, we integrate sensing technology (i.e., eye-tracking) and virtual reality (VR) to model problem-solving in manufacturing systems. A novel data-driven model that integrates signal detection theory (SDT) with Conflict & Error (C&E) is developed to quantify engineering problem-solving skills. First, we simulate a manufacturing system in a VR game environment. Students are given an assembly problem to produce a car toy that satisfies some particular requirements in the VR manufacturing system while eye-tracking data are collected throughout the assembly process. Second, eye-tracking data are analyzed with a SDT model to quantify problem-solving skills. Third, a joint SDT-C&E model is developed to analyze eye-tracking data and benchmark with results generated from the SDT model. Experimental results show that the joint SDT-C&E model is more effective to quantify problem-solving skills of engineering students than the SDT model.