TY - JOUR
T1 - Using the analytic hierarchy process to examine judgment consistency in a complex multiattribute task
AU - Jin, Jeonghwan
AU - Rothrock, Ling
AU - McDermott, Patricia L.
AU - Barnes, Michael
N1 - Funding Information:
Manuscript received May 28, 2008; revised May 3, 2009. Date of publication April 26, 2010; date of current version August 18, 2010. This work was supported in part by the U.S. Army Research Laboratory and in part by Alion Science and Technology. The views, opinions, and/or findings expressed in this paper are those of the author(s) and should not be construed as an official Department of the Army position, policy, or decision, unless so designated by other documentation. This paper was recommended by Associate Editor E. J. Bass.
PY - 2010/9
Y1 - 2010/9
N2 - This paper investigates the impact of framing and time pressure on human judgment performance in a complex multiattribute judgment task. We focus on the decision process of human participants who must choose between pairwise alternatives in a resource-allocation task. We used the Analytic Hierarchy Process (AHP) to calculate the relative weights of the four alternatives (i.e., C1, C2, C3, and C4) and the judgment consistency. Using the AHP, we examined two sets of hypotheses that address the impact of task conditions on the weight prioritization of choice alternatives and the internal consistency of the judgment behavior under varying task conditions. The experiment simulated the allocation of robotic assets across the battlefield to collect data about an enemy. Participants had to make a judgment about which asset to allocate to a new area by taking into account three criteria related to the likelihood of success. We manipulated the information frame and the nature of the task. We found that, in general, participants gave significantly different weights to the same alternatives under different frames and task conditions. Specifically, in terms of ln-transformed priority weights, participants gave significantly lower weights to C2 and C4 and higher weight to C3 under gain frame than under loss frame, and also, under different task conditions (i.e., Tasks #1, #2, and #3), participants gave significantly higher weight to C4 in Task #1, lower weights to C1 and C4, higher weight to C3 in Task #2, and lower weight to C3 in Task #3. Furthermore, we found that the internal consistency of the decision behavior was worse, first, in the loss frame than the gain frame and, second, under time pressure. Our methodology complements utility-theoretic frameworks by assessing judgment consistency without requiring the use of task-performance outcomes. This work is a step toward establishing a coherence criterion to investigate judgment under naturalistic conditions. The results will be useful for the design of multiattribute interfaces and decision aiding tools for real-time judgments in time-pressured task environments.
AB - This paper investigates the impact of framing and time pressure on human judgment performance in a complex multiattribute judgment task. We focus on the decision process of human participants who must choose between pairwise alternatives in a resource-allocation task. We used the Analytic Hierarchy Process (AHP) to calculate the relative weights of the four alternatives (i.e., C1, C2, C3, and C4) and the judgment consistency. Using the AHP, we examined two sets of hypotheses that address the impact of task conditions on the weight prioritization of choice alternatives and the internal consistency of the judgment behavior under varying task conditions. The experiment simulated the allocation of robotic assets across the battlefield to collect data about an enemy. Participants had to make a judgment about which asset to allocate to a new area by taking into account three criteria related to the likelihood of success. We manipulated the information frame and the nature of the task. We found that, in general, participants gave significantly different weights to the same alternatives under different frames and task conditions. Specifically, in terms of ln-transformed priority weights, participants gave significantly lower weights to C2 and C4 and higher weight to C3 under gain frame than under loss frame, and also, under different task conditions (i.e., Tasks #1, #2, and #3), participants gave significantly higher weight to C4 in Task #1, lower weights to C1 and C4, higher weight to C3 in Task #2, and lower weight to C3 in Task #3. Furthermore, we found that the internal consistency of the decision behavior was worse, first, in the loss frame than the gain frame and, second, under time pressure. Our methodology complements utility-theoretic frameworks by assessing judgment consistency without requiring the use of task-performance outcomes. This work is a step toward establishing a coherence criterion to investigate judgment under naturalistic conditions. The results will be useful for the design of multiattribute interfaces and decision aiding tools for real-time judgments in time-pressured task environments.
UR - http://www.scopus.com/inward/record.url?scp=77955848669&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=77955848669&partnerID=8YFLogxK
U2 - 10.1109/TSMCA.2010.2045119
DO - 10.1109/TSMCA.2010.2045119
M3 - Article
AN - SCOPUS:77955848669
SN - 1083-4427
VL - 40
SP - 1105
EP - 1115
JO - IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans
JF - IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans
IS - 5
M1 - 5454346
ER -