Multimodal Hidden Markov Models for Real-Time Human Proficiency Assessment in Industry 5.0: Integrating Physiological, Behavioral, and Subjective Metrics

This paper presents a Multimodal Hidden Markov Model (MHMM) framework specifically designed for real-time human proficiency assessment, integrating physiological (Heart Rate Variability (HRV)), behavioral (Task Completion Time (TCT)), and subjective (NASA Task Load Index (NASA-TLX)) data streams to infer latent human proficiency states in industrial settings. Using published empirical data from the surgical training literature, a comprehensive simulation study was conducted, with the MHMM (Trained) achieving 92.5% classification accuracy, significantly outperforming unimodal Hidden Markov Model (HMM) variants 61–63.9% and demonstrating competitive performance with advanced models such as Long Short-Term Memory (LSTM) networks 90%, and Conditional Random Field (CRF) 88.5%. The framework exhibited robustness across stress-test scenarios, including sensor noise, missing data, and imbalanced class distributions. A key advantage of the MHMM over black-box approaches is its interpretability by providing quantifiable transition probabilities that reveal learning rates, forgetting patterns, and contextual influences on proficiency dynamics. The model successfully captures context-dependent effects, including task complexity and cumulative fatigue, through dynamic transition matrices. When demonstrated through simulation, this framework establishes a foundation for developing adaptive operator-AI collaboration systems in Industry 5.0 environments. The MHMM’s combination of high accuracy, robustness, and interpretability makes it a promising candidate for future empirical validation in real-world industrial, healthcare, and training applications in which it is critical to understand and support human proficiency development.