Sensor-based modeling of problem-solving in virtual reality manufacturing systems

Rui Zhu, Faisal Aqlan, Richard Zhao, Hui Yang

Research output: Contribution to journalArticlepeer-review

6 Scopus citations


Problem-solving is the process to achieve a goal when the solution path is uncertain. Recently, technological advancements have changed problems’ characteristics and their solutions in engineering fields. Strong problem-solving skills are essential to allow engineers to assess new problems and quickly implement solutions. Engineering problem-solving skills are first educated in schools and usually evaluated through written exams. However, high grades in exams do not represent sufficient problem-solving skills in real-world engineering problems. Decision making with insufficient problem-solving skills in real world may result in costly consequences. Therefore, it is imperative to evaluate and reinforce problem-solving skills of engineering students in real-world problems. With the rapid technological advancements, availability of virtual reality (VR) and eye-tracking facilitates the study of engineering problem-solving. The immersive environment created by VR enables students to better understand and solve real-world engineering problems. On the other hand, eye-tracking allows for studying fundamental cognitive processes during information processing. It is critical to integrate VR simulation with data-driven modeling of eye movements to evaluate and enhance engineering problem-solving skills. In this paper, we integrate sensing technology (i.e., eye-tracking) and virtual reality (VR) to model problem-solving in manufacturing systems. A novel data-driven model that integrates signal detection theory (SDT) with Conflict & Error (C&E) is developed to quantify engineering problem-solving skills. First, we simulate a manufacturing system in a VR game environment. Students are given an assembly problem to produce a car toy that satisfies some particular requirements in the VR manufacturing system while eye-tracking data are collected throughout the assembly process. Second, eye-tracking data are analyzed with a SDT model to quantify problem-solving skills. Third, a joint SDT-C&E model is developed to analyze eye-tracking data and benchmark with results generated from the SDT model. Experimental results show that the joint SDT-C&E model is more effective to quantify problem-solving skills of engineering students than the SDT model.

Original languageEnglish (US)
Article number117220
JournalExpert Systems With Applications
StatePublished - Sep 1 2022

All Science Journal Classification (ASJC) codes

  • General Engineering
  • Computer Science Applications
  • Artificial Intelligence


Dive into the research topics of 'Sensor-based modeling of problem-solving in virtual reality manufacturing systems'. Together they form a unique fingerprint.

Cite this