Combining hidden Markov model and fuzzy neural network for continuous recognition of complex dynamic gestures

In the design of gesture-based user interfaces, continuously recognizing complex dynamic gestures is a challenging task, because of the high-dimensional information of gestures, ambiguous semantic meanings of gestures, and the presence of unpredictable non-gesture body motions. In this paper, we propose a hybrid model that can leverage the time-series modeling ability of hidden Markov model and the fuzzy inference ability of fuzzy neural network. First, a complex dynamic gesture is decomposed and fed into the hybrid model. The likelihood probability of an observation sequence estimated by the hidden Markov model is used as fuzzy membership degree of the corresponding fuzzy class variable in fuzzy neural network. Next, fuzzy rule modeling and fuzzy inference are performed by fuzzy neural network for gesture classification. To spot key gestures accurately, a threshold model is introduced to calculate the likelihood threshold of an input pattern and provide a reliability measure of whether to accept the pattern as a gesture. Finally, the proposed method is applied to recognize ten user-defined dynamic gestures for controlling interactive digital television in a smart room. Results of our experiment show that the proposed method performed better in terms of spotting reliability and recognition accuracy than conventional gesture recognition methods.