Predicting Mouse Click Position Using Long Short-Term Memory Model Trained by Joint Loss Function

Datong Wei; Chaofan Yang; Xiaolong (Luke) Zhang; Xiaoru Yuan

doi:10.1145/3411763.3451651

Predicting Mouse Click Position Using Long Short-Term Memory Model Trained by Joint Loss Function

Datong Wei
, Chaofan Yang
, Xiaolong (Luke) Zhang
, Xiaoru Yuan

College of Information Sciences and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

6 Scopus citations

Abstract

Knowing where users might click in advance can potentially improve the efficiency of user interaction in desktop user interfaces. In this paper, we propose a machine learning approach to predict mouse click location. Our model, which is LSTM (long short-term memory)-based and trained by joint supervision, can predict the rectangular region of mouse click with feeding mouse trajectories on the fly. Experiment results show that our model can achieve a result of a predicted rectangle area of 58 × 79 pixels with 92% accuracy, and reduce prediction error when compared with other state-of-the-art prediction methods using a multi-user dataset.

Original language	English (US)
Title of host publication	Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA 2021
Publisher	Association for Computing Machinery
ISBN (Electronic)	9781450380959
DOIs	https://doi.org/10.1145/3411763.3451651
State	Published - May 8 2021
Event	2021 CHI Conference on Human Factors in Computing Systems: Making Waves, Combining Strengths, CHI EA 2021 - Virtual, Online, Japan Duration: May 8 2021 → May 13 2021

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Conference

Conference	2021 CHI Conference on Human Factors in Computing Systems: Making Waves, Combining Strengths, CHI EA 2021
Country/Territory	Japan
City	Virtual, Online
Period	5/8/21 → 5/13/21

All Science Journal Classification (ASJC) codes

Software
Human-Computer Interaction
Computer Graphics and Computer-Aided Design

Access to Document

10.1145/3411763.3451651

Cite this

Wei, D., Yang, C., Zhang, X., & Yuan, X. (2021). Predicting Mouse Click Position Using Long Short-Term Memory Model Trained by Joint Loss Function. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA 2021 (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery. https://doi.org/10.1145/3411763.3451651

@inproceedings{e654ea9f1ed84a2e81452dba5f8a8b87,

title = "Predicting Mouse Click Position Using Long Short-Term Memory Model Trained by Joint Loss Function",

abstract = "Knowing where users might click in advance can potentially improve the efficiency of user interaction in desktop user interfaces. In this paper, we propose a machine learning approach to predict mouse click location. Our model, which is LSTM (long short-term memory)-based and trained by joint supervision, can predict the rectangular region of mouse click with feeding mouse trajectories on the fly. Experiment results show that our model can achieve a result of a predicted rectangle area of 58 × 79 pixels with 92\% accuracy, and reduce prediction error when compared with other state-of-the-art prediction methods using a multi-user dataset.",

author = "Datong Wei and Chaofan Yang and Zhang, \{Xiaolong (Luke)\} and Xiaoru Yuan",

note = "Publisher Copyright: {\textcopyright} 2021 ACM.; 2021 CHI Conference on Human Factors in Computing Systems: Making Waves, Combining Strengths, CHI EA 2021 ; Conference date: 08-05-2021 Through 13-05-2021",

year = "2021",

month = may,

day = "8",

doi = "10.1145/3411763.3451651",

language = "English (US)",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery",

booktitle = "Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA 2021",

}

Wei, D, Yang, C, Zhang, X & Yuan, X 2021, Predicting Mouse Click Position Using Long Short-Term Memory Model Trained by Joint Loss Function. in Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA 2021. Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, 2021 CHI Conference on Human Factors in Computing Systems: Making Waves, Combining Strengths, CHI EA 2021, Virtual, Online, Japan, 5/8/21. https://doi.org/10.1145/3411763.3451651

Predicting Mouse Click Position Using Long Short-Term Memory Model Trained by Joint Loss Function. / Wei, Datong; Yang, Chaofan; Zhang, Xiaolong (Luke) et al.
Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA 2021. Association for Computing Machinery, 2021. (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Predicting Mouse Click Position Using Long Short-Term Memory Model Trained by Joint Loss Function

AU - Wei, Datong

AU - Yang, Chaofan

AU - Zhang, Xiaolong (Luke)

AU - Yuan, Xiaoru

PY - 2021/5/8

Y1 - 2021/5/8

N2 - Knowing where users might click in advance can potentially improve the efficiency of user interaction in desktop user interfaces. In this paper, we propose a machine learning approach to predict mouse click location. Our model, which is LSTM (long short-term memory)-based and trained by joint supervision, can predict the rectangular region of mouse click with feeding mouse trajectories on the fly. Experiment results show that our model can achieve a result of a predicted rectangle area of 58 × 79 pixels with 92% accuracy, and reduce prediction error when compared with other state-of-the-art prediction methods using a multi-user dataset.

AB - Knowing where users might click in advance can potentially improve the efficiency of user interaction in desktop user interfaces. In this paper, we propose a machine learning approach to predict mouse click location. Our model, which is LSTM (long short-term memory)-based and trained by joint supervision, can predict the rectangular region of mouse click with feeding mouse trajectories on the fly. Experiment results show that our model can achieve a result of a predicted rectangle area of 58 × 79 pixels with 92% accuracy, and reduce prediction error when compared with other state-of-the-art prediction methods using a multi-user dataset.

UR - https://www.scopus.com/pages/publications/85105824641

UR - https://www.scopus.com/pages/publications/85105824641#tab=citedBy

U2 - 10.1145/3411763.3451651

DO - 10.1145/3411763.3451651

M3 - Conference contribution

AN - SCOPUS:85105824641

T3 - Conference on Human Factors in Computing Systems - Proceedings

BT - Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA 2021

PB - Association for Computing Machinery

T2 - 2021 CHI Conference on Human Factors in Computing Systems: Making Waves, Combining Strengths, CHI EA 2021

Y2 - 8 May 2021 through 13 May 2021

ER -

Predicting Mouse Click Position Using Long Short-Term Memory Model Trained by Joint Loss Function

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this