Utilization of Automatized Creativity Ratings in Linguistically Diverse Populations: Automated Scores Align with Human Ratings

Measuring an individual's creativity typically relies on labor-intensive subjective ratings of the quality of ideas and solutions to problems. In the Alternate Uses Task (AUT), frequently used in engineering design education for concept generation and to gauge creative function-object relationships, participants generate as many novel uses of everyday objects as possible within a given time frame. Unfortunately, objective and rapid evaluation of AUT responses for levels of originality and usefulness is difficult. Recently, an automatized method for generating scores has been developed, the freely accessible Semantic Distance (SemDis) tool [1]. Given the linguistic and cultural diversity of engineering students in the U.S., it seems fair to question how well this type of automatic rating system, based on prototypical language models, captures the creativity of engineering students who may be nonnative speakers of English. We extensively trained human raters to score the AUT responses of multilingual engineering students living in either a non-English environment or in the US, and the AUT responses of monolingual English engineering students. We found that the human ratings of all three groups of engineering students correlated strongly, and positively, with the automatic SemDis ratings. This forms proof of concept for using automatic rating systems such as SemDis in engineering classroom settings. In addition to saving evaluators' time, this method may also be preferred because it is unbiased to cultural and linguistic features of responders' answers that might reveal their gender, race, ethnic or linguistic background information.