TY - JOUR
T1 - Offline Evaluation Matters
T2 - Investigation of the Influence of Offline Performance on Real-Time Operation of Electromyography-Based Neural-Machine Interfaces
AU - Hinson, Robert M.
AU - Berman, Joseph
AU - Filer, William
AU - Kamper, Derek
AU - Hu, Xiaogang
AU - Huang, He
N1 - Publisher Copyright:
© 2001-2011 IEEE.
PY - 2023
Y1 - 2023
N2 - There has been a debate on the most appropriate way to evaluate electromyography (EMG)-based neural-machine interfaces (NMIs). Accordingly, this study examined whether a relationship between offline kinematic predictive accuracy (R2) and user real-time task performance while using the interface could be identified. A virtual posture-matching task was developed to evaluate motion capture-based control and myoelectric control with artificial neural networks (ANNs) trained to low (R2 ≈ 0.4), moderate (R2 ≈ 0.6), and high (RvphantomD a 2 0.8) offline performance levels. Twelve non-disabled subjects trained with each offline performance level decoder before evaluating final real-time posture matching performance. Moderate to strong relationships were detected between offline performance and all real-time task performance metrics: task completion percentage (r = 0.66, p < 0.001), normalized task completion time (r =-0.51, p = 0.001), path efficiency (r = 0.74, p < 0.001), and target overshoots (r =-0.79, p < 0.001). Significant improvements in each real-time task evaluation metric were also observed between the different offline performance levels. Additionally, subjects rated myoelectric controllers with higher offline performance more favorably. The results of this study support the use and validity of offline analyses for optimization of NMIs in myoelectric control research and development.
AB - There has been a debate on the most appropriate way to evaluate electromyography (EMG)-based neural-machine interfaces (NMIs). Accordingly, this study examined whether a relationship between offline kinematic predictive accuracy (R2) and user real-time task performance while using the interface could be identified. A virtual posture-matching task was developed to evaluate motion capture-based control and myoelectric control with artificial neural networks (ANNs) trained to low (R2 ≈ 0.4), moderate (R2 ≈ 0.6), and high (RvphantomD a 2 0.8) offline performance levels. Twelve non-disabled subjects trained with each offline performance level decoder before evaluating final real-time posture matching performance. Moderate to strong relationships were detected between offline performance and all real-time task performance metrics: task completion percentage (r = 0.66, p < 0.001), normalized task completion time (r =-0.51, p = 0.001), path efficiency (r = 0.74, p < 0.001), and target overshoots (r =-0.79, p < 0.001). Significant improvements in each real-time task evaluation metric were also observed between the different offline performance levels. Additionally, subjects rated myoelectric controllers with higher offline performance more favorably. The results of this study support the use and validity of offline analyses for optimization of NMIs in myoelectric control research and development.
UR - http://www.scopus.com/inward/record.url?scp=85144812628&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85144812628&partnerID=8YFLogxK
U2 - 10.1109/TNSRE.2022.3226229
DO - 10.1109/TNSRE.2022.3226229
M3 - Article
C2 - 37015358
AN - SCOPUS:85144812628
SN - 1534-4320
VL - 31
SP - 680
EP - 689
JO - IEEE Transactions on Neural Systems and Rehabilitation Engineering
JF - IEEE Transactions on Neural Systems and Rehabilitation Engineering
ER -