Should an Analyst Share Calibration Information with Experts?

Many times, expert judgment is used in a two-step procedure: (i) obtain judgments for calibration quantities for which true/empirical values are available to calibrate the expert’s judgmental errors, and then (ii) obtain judgments for focal quantities (quantities of interest but without historical data), adjust them using the calibration information, and then use the adjusted judgments to assist decision making. In such situations, should a decision analyst share the calibration information with the expert before the expert provides judgments for focal quantities? We answer this question using laboratory experiments. We specifically investigate the role of task complexity, numeracy, and self-awareness on the use of calibration information by the expert. We find that: (a) Expert judgment tends to be of a worse quality as task complexity increases, (b) Calibration feedback does not improve managerial judgments in a less complex task, but it does reduce the bias and (especially) noise in a more complex task, (c) Numeracy does not impact the use of calibration information by experts regardless of task complexity, and (d) Individuals are able to directionally discern whether they are doing well in less complex tasks, but not in more complex tasks. As such these results suggest that when faced with complex tasks experts do benefit from receiving calibration information. However, this benefit comes at the expense of rendering this information inapplicable for the decision analyst to adjust the focal judgments provided by an expert. In contrast, experts do not benefit from receiving calibration information for simple tasks, allowing decision analysts to continue to use the calibration information to adjust managerial judgments.