TY - JOUR
T1 - Predictive aids can lead to sustained attention decrements in the detection of non-routine critical events in event monitoring
AU - Minotra, Dev
AU - McNeese, Michael D.
N1 - Funding Information:
This work was partially supported by US Army Research Office (ARO) MURI Grant 'Computer Aided Human Centric Cyber Situation Awareness' W911-NF-09-1-0525. We would like to thank members of the MINDS Lab at Penn State for their support. We would like to thank the reviewers for their insightful comments. Additionally, the support and input from Dr. Catherine Burns, Dr. Xiaolong Zhang, and Dr. Ling Rothrock are also highly appreciated.
Publisher Copyright:
© 2017, Springer-Verlag London.
PY - 2017/2/1
Y1 - 2017/2/1
N2 - Task performance in uncertain and complex environments is dependent on human effectiveness in dynamic prioritisation and attention allocation. Non-routine critical events place a high demand on cognitive resources in operators, and they require operators to dynamically re-prioritise sub-tasks calling upon the need for predictive aids. Predictive aids provide real-time prediction of system variables, and they can potentially facilitate the detection of non-routine critical events and dynamic re-prioritisation during such events. However, prior research has reported mixed findings in their effectiveness, and no prior research has been done to test the effectiveness of predictive aids during non-routine critical events that require operators to dynamically prioritise tasks. Our experimental study with 77 participants examined the effect of a predictive aid on prioritisation of non-routine critical events in a cyber security event monitoring task. The predictive aid resulted in decrements in sustained attention as seen in delayed detection of some non-routine critical events and errors in prioritising non-routine critical events. This effect is a result of miscalibration in the importance of looking ahead with the predictive aid. This miscalibration was possibly associated with the look-ahead time, complexity of the predictive aid, and the lack of automation transparency (i.e. mechanism underlying prediction) together affecting its perceived usefulness. The experimental results have the following implications on the design and testing of predictive aids: (1) in a task requiring dynamic prioritisation and detection of non-routine critical events, predictive aids with insufficient look-ahead times can result in decrements in sustained attention; (2) predictive aids should be evaluated with subjective measures of perceived usefulness and workload; (3) predictive aids should be designed to have sufficient look-ahead times and transparency of underlying prediction mechanisms as these would affect visual scanning effort and perceived usefulness; and (4) experimental tests of their effectiveness should involve scenarios long enough that would take sustained attention effects and any potential learning effects into consideration.
AB - Task performance in uncertain and complex environments is dependent on human effectiveness in dynamic prioritisation and attention allocation. Non-routine critical events place a high demand on cognitive resources in operators, and they require operators to dynamically re-prioritise sub-tasks calling upon the need for predictive aids. Predictive aids provide real-time prediction of system variables, and they can potentially facilitate the detection of non-routine critical events and dynamic re-prioritisation during such events. However, prior research has reported mixed findings in their effectiveness, and no prior research has been done to test the effectiveness of predictive aids during non-routine critical events that require operators to dynamically prioritise tasks. Our experimental study with 77 participants examined the effect of a predictive aid on prioritisation of non-routine critical events in a cyber security event monitoring task. The predictive aid resulted in decrements in sustained attention as seen in delayed detection of some non-routine critical events and errors in prioritising non-routine critical events. This effect is a result of miscalibration in the importance of looking ahead with the predictive aid. This miscalibration was possibly associated with the look-ahead time, complexity of the predictive aid, and the lack of automation transparency (i.e. mechanism underlying prediction) together affecting its perceived usefulness. The experimental results have the following implications on the design and testing of predictive aids: (1) in a task requiring dynamic prioritisation and detection of non-routine critical events, predictive aids with insufficient look-ahead times can result in decrements in sustained attention; (2) predictive aids should be evaluated with subjective measures of perceived usefulness and workload; (3) predictive aids should be designed to have sufficient look-ahead times and transparency of underlying prediction mechanisms as these would affect visual scanning effort and perceived usefulness; and (4) experimental tests of their effectiveness should involve scenarios long enough that would take sustained attention effects and any potential learning effects into consideration.
UR - http://www.scopus.com/inward/record.url?scp=85010931291&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85010931291&partnerID=8YFLogxK
U2 - 10.1007/s10111-017-0402-x
DO - 10.1007/s10111-017-0402-x
M3 - Article
AN - SCOPUS:85010931291
SN - 1435-5558
VL - 19
SP - 161
EP - 177
JO - Cognition, Technology and Work
JF - Cognition, Technology and Work
IS - 1
ER -