A Practical System for 3-D Hand Pose Tracking Using EMG Wearables With Applications to Prosthetics and User Interfaces

Yilin Liu, Shijia Zhang, Mahanth Gowda

Research output: Contribution to journalArticlepeer-review

4 Scopus citations

Abstract

Ubiquitous finger motion tracking enables a number of exciting applications in augmented reality, sports analytics, rehabilitation-healthcare, haptics, etc. This article presents NeuroPose, a system that shows the feasibility of 3-D finger motion tracking using a platform of wearable electromyography (EMG) sensors. EMG sensors can sense electrical potential from muscles due to finger activation, thus offering rich information for fine-grained finger motion sensing. However, converting the sensor information to 3-D finger poses is non trivial since signals from multiple fingers superimpose at the sensor in complex patterns. Toward solving this problem, NeuroPose fuses information from anatomical constraints of finger motion with machine learning architectures on recurrent neural networks (RNNs), encoder-decoder networks, and ResNets to extract 3-D finger motion from noisy EMG data. The generated motion pattern is temporally smooth as well as anatomically consistent. Furthermore, a transfer learning algorithm is leveraged to adapt a pretrained model on one user to a new user with minimal training overhead. A systematic study with 12 users demonstrates a median error of 6.24° and a 90%-ile error of 18.33° in tracking 3-D finger joint angles. The accuracy is robust to natural variation in sensor mounting positions as well as changes in wrist positions of the user. In addition, this article validates the feasibility of mirrored bilateral training approach with applications in prosthetic devices. Finally, NeuroPose is comprehensively evaluated on both low-end and recent smartphones with a processing latency of 0.019 s and low energy overhead.

Original languageEnglish (US)
Pages (from-to)3407-3427
Number of pages21
JournalIEEE Internet of Things Journal
Volume10
Issue number4
DOIs
StatePublished - Feb 15 2023

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Information Systems
  • Hardware and Architecture
  • Computer Science Applications
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'A Practical System for 3-D Hand Pose Tracking Using EMG Wearables With Applications to Prosthetics and User Interfaces'. Together they form a unique fingerprint.

Cite this