HEY, AI! CAN YOU SEE WHAT I SEE? MULTIMODAL TRANSFER LEARNING-BASED DESIGN METRICS PREDICTION FOR SKETCHES WITH TEXT DESCRIPTIONS

Measuring design creativity is an indispensable component of innovation in engineering design. Properly assessing the creativity of a design requires a rigorous evaluation of the outputs. Traditional methods to evaluate designs are slow, expensive, and difficult to scale because they rely on human expert input. An alternative approach is to use computational methods to evaluate designs. However, most existing methods have limited utility because they are constrained to unimodal design representations (e.g., texts or sketches) and small datasets. To overcome these limitations, we propose a multimodal transfer learning-based machine learning model to predict five design metrics: drawing quality, uniqueness, elegance, usefulness, and creativity. The proposed model utilizes knowledge from large external datasets through transfer learning and simultaneously processes text and sketch data from early-phase concepts through multimodal learning. Through six unimodal models using only texts or sketches, we show that transfer learning improves the predictive validity of text learning and sketch learning by 2%-18% and 9%-24%, respectively, for design metric evaluation. By comparing our multimodal model with the best unimodal models, we demonstrate that joining unimodal text and sketch learning models further increases the predictive validity of the approach by 4%-10%. The proposed models are generalizable to many application contexts beyond design concepts. Our findings highlight the importance of analyzing designs from multiple perspectives for design assessment. Finally, we discuss the challenges and opportunities in developing AI models for design metric evaluation.