Using Java to provide cognitive models with a more universal way to interact with graphical user interfaces

Farnaz Tehranchi; Frank E. Ritter

Using Java to provide cognitive models with a more universal way to interact with graphical user interfaces

Research output: Contribution to conference › Paper › peer-review

Abstract

We present JSegMan, an approach to extend interaction in the ACTR cognitive architecture. JSegMan allows cognitive models to interact directlywith any application on a PC. JSegMan also simulates human visual attentionmovements. This work allows the behavior of models to be compared more directly to human behavior by using the same uninstrumented interface. Thiswork also provides direct support for automated user interface testing andclosed-loop system control. Furthermore, a new data structure-visual patterns-has been introduced to provide a more realistic representation of theworld. We tested JSegMan by using it with an existing ACT-R model of aspreadsheet task, the Dismal model. Because the model interacted directly witha spreadsheet, we found defects in the Dismal model and resolved them. Therevised model more accurately predicted a 20-min task while performing thetask.

Original language	English (US)
State	Published - Jan 1 2018
Event	2018 International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction and Behavior Representation in Modeling and Simulation, BRiMS 2018 - Washington, United States Duration: Jul 10 2018 → Jul 13 2018

Conference

Conference	2018 International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction and Behavior Representation in Modeling and Simulation, BRiMS 2018
Country/Territory	United States
City	Washington
Period	7/10/18 → 7/13/18

All Science Journal Classification (ASJC) codes

Human-Computer Interaction
Modeling and Simulation

Cite this

@conference{5199a68d91024362a02e02a4984d0094,

title = "Using Java to provide cognitive models with a more universal way to interact with graphical user interfaces",

abstract = "We present JSegMan, an approach to extend interaction in the ACTR cognitive architecture. JSegMan allows cognitive models to interact directlywith any application on a PC. JSegMan also simulates human visual attentionmovements. This work allows the behavior of models to be compared more directly to human behavior by using the same uninstrumented interface. Thiswork also provides direct support for automated user interface testing andclosed-loop system control. Furthermore, a new data structure-visual patterns-has been introduced to provide a more realistic representation of theworld. We tested JSegMan by using it with an existing ACT-R model of aspreadsheet task, the Dismal model. Because the model interacted directly witha spreadsheet, we found defects in the Dismal model and resolved them. Therevised model more accurately predicted a 20-min task while performing thetask.",

author = "Farnaz Tehranchi and Ritter, {Frank E.}",

year = "2018",

month = jan,

day = "1",

language = "English (US)",

note = "2018 International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction and Behavior Representation in Modeling and Simulation, BRiMS 2018 ; Conference date: 10-07-2018 Through 13-07-2018",

}

Tehranchi, F & Ritter, FE 2018, 'Using Java to provide cognitive models with a more universal way to interact with graphical user interfaces', Paper presented at 2018 International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction and Behavior Representation in Modeling and Simulation, BRiMS 2018, Washington, United States, 7/10/18 - 7/13/18.

Using Java to provide cognitive models with a more universal way to interact with graphical user interfaces. / Tehranchi, Farnaz ; Ritter, Frank E.
2018. Paper presented at 2018 International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction and Behavior Representation in Modeling and Simulation, BRiMS 2018, Washington, United States.

Research output: Contribution to conference › Paper › peer-review

TY - CONF

T1 - Using Java to provide cognitive models with a more universal way to interact with graphical user interfaces

AU - Tehranchi, Farnaz

AU - Ritter, Frank E.

PY - 2018/1/1

Y1 - 2018/1/1

N2 - We present JSegMan, an approach to extend interaction in the ACTR cognitive architecture. JSegMan allows cognitive models to interact directlywith any application on a PC. JSegMan also simulates human visual attentionmovements. This work allows the behavior of models to be compared more directly to human behavior by using the same uninstrumented interface. Thiswork also provides direct support for automated user interface testing andclosed-loop system control. Furthermore, a new data structure-visual patterns-has been introduced to provide a more realistic representation of theworld. We tested JSegMan by using it with an existing ACT-R model of aspreadsheet task, the Dismal model. Because the model interacted directly witha spreadsheet, we found defects in the Dismal model and resolved them. Therevised model more accurately predicted a 20-min task while performing thetask.

AB - We present JSegMan, an approach to extend interaction in the ACTR cognitive architecture. JSegMan allows cognitive models to interact directlywith any application on a PC. JSegMan also simulates human visual attentionmovements. This work allows the behavior of models to be compared more directly to human behavior by using the same uninstrumented interface. Thiswork also provides direct support for automated user interface testing andclosed-loop system control. Furthermore, a new data structure-visual patterns-has been introduced to provide a more realistic representation of theworld. We tested JSegMan by using it with an existing ACT-R model of aspreadsheet task, the Dismal model. Because the model interacted directly witha spreadsheet, we found defects in the Dismal model and resolved them. Therevised model more accurately predicted a 20-min task while performing thetask.

UR - http://www.scopus.com/inward/record.url?scp=85075725282&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85075725282&partnerID=8YFLogxK

M3 - Paper

T2 - 2018 International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction and Behavior Representation in Modeling and Simulation, BRiMS 2018

Y2 - 10 July 2018 through 13 July 2018

ER -

Using Java to provide cognitive models with a more universal way to interact with graphical user interfaces

Abstract

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this