Towards the development of an objective assessment technique for use in engineering design education

Design is now considered to be the central or distinguishing activity of engineering and thus is being widely integrated throughout engineering education. However, engineering programs face special challenges in assessing student design abilities because of the subjective nature of design. Research has shown that current evaluation methods are insufficient in evaluating student design performance because quantitative tests of students design skills are not indicators of students design performance and open-ended project evaluations are mudded by the subjective biases of the single human evaluator. The purpose of this paper is to evaluate a new evaluation metric that utilizes a novel application of the Bayesian Truth Serum (BTS) in an effort to increase the accuracy of design skill evaluations and minimize subjective bias. Through a series of experiments our results indicate that BTS is a better indicator of idea generation ability than traditional evaluation methods. Although more work is needed to validate and understand how BTS can be used as an evaluation technique in engineering design, the results from this study are promising and provide a first step at deriving quantitative measures of student design performance in engineering.