TY - JOUR
T1 - Semantic Spaces Are Not Created Equal - How Should We Weigh Them in the Sequel?
T2 - On Composites in Automated Creativity Scoring
AU - Forthmann, Boris
AU - Beaty, Roger E.
AU - Johnson, Dan R.
N1 - Publisher Copyright:
© 2022 The Author(s).
PY - 2023/11
Y1 - 2023/11
N2 - Semantic distance scoring provides an attractive alternative to other scoring approaches for responses in creative thinking tasks. In addition, evidence in support of semantic distance scoring has increased over the last few years. In one recent approach, it has been proposed to combine multiple semantic spaces to better balance the idiosyncratic influences of each space. Thereby, final semantic distance scores for each response are represented by a composite or factor score. However, semantic spaces are not necessarily equally weighted in mean scores, and the usage of factor scores requires high levels of factor determinacy (i.e., the correlation between estimates and true factor scores). Hence, in this work, we examined the weighting underlying mean scores, mean scores of standardized variables, factor loadings, weights that maximize reliability, and equally effective weights on common verbal creative thinking tasks. Both empirical and simulated factor determinacy, as well as Gilmer-Feldt's composite reliability, were mostly good to excellent (i.e., > .80) across two task types (Alternate Uses and Creative Word Association), eight samples of data, and all weighting approaches. Person-level validity findings were further highly comparable across weighting approaches. Observed nuances and challenges of different weightings and the question of using composites vs. factor scores are thoroughly provided.
AB - Semantic distance scoring provides an attractive alternative to other scoring approaches for responses in creative thinking tasks. In addition, evidence in support of semantic distance scoring has increased over the last few years. In one recent approach, it has been proposed to combine multiple semantic spaces to better balance the idiosyncratic influences of each space. Thereby, final semantic distance scores for each response are represented by a composite or factor score. However, semantic spaces are not necessarily equally weighted in mean scores, and the usage of factor scores requires high levels of factor determinacy (i.e., the correlation between estimates and true factor scores). Hence, in this work, we examined the weighting underlying mean scores, mean scores of standardized variables, factor loadings, weights that maximize reliability, and equally effective weights on common verbal creative thinking tasks. Both empirical and simulated factor determinacy, as well as Gilmer-Feldt's composite reliability, were mostly good to excellent (i.e., > .80) across two task types (Alternate Uses and Creative Word Association), eight samples of data, and all weighting approaches. Person-level validity findings were further highly comparable across weighting approaches. Observed nuances and challenges of different weightings and the question of using composites vs. factor scores are thoroughly provided.
UR - http://www.scopus.com/inward/record.url?scp=85135022648&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85135022648&partnerID=8YFLogxK
U2 - 10.1027/1015-5759/a000723
DO - 10.1027/1015-5759/a000723
M3 - Article
AN - SCOPUS:85135022648
SN - 1015-5759
VL - 39
SP - 449
EP - 461
JO - European Journal of Psychological Assessment
JF - European Journal of Psychological Assessment
IS - 6
ER -