Studying the reliability and validity of test scores for mathematical and spatial reasoning tasks for engineering students

While preliminary, we interpret our results as encouraging as we attempt to measure four different constructs that we believe are keys to academic success in engineering. Two constructs represent mathematical skills or traits. Two constructs represent spatial reasoning strategies or traits. Correlation coefficients indicated there were some moderate associations between pairs of constructs (e.g., M1 and S2), and a preliminary factor analysis showed that part of the relation between constructs may be related to the types of external representations (e.g., diagrams, equations, graphs) presented in the test items or the primary physics concepts (e.g. acceleration, force, motion) for which the items have been designed to assess achievement in engineering. In the future, we will plan studies so that more items are included on each scale. Further, we will attempt to replicate the initial findings reported in this manuscript. If correlation coefficients between scores are at best moderate, then this pattern of results has important implications for assessment, teaching, and research in engineering education. Simply, it implies that the skill set required to succeed in engineering may be multidimensional. As such, a set of various tasks or tests are needed to help students understand their profiles of strengths and weaknesses as they attempt to meet degree requirements in this, sometimes, challenging field. In future research, it might be important to study how students think aloud as they solve problems not only within one construct set, but also, how they compare and contrast the problems they attempt to solve across sets. A mixed-methodological approach where we study both test scores with verbal reports might help us understand more about the complex processing required to complete exercises successfully. Further, verbal protocol summaries might reveal the role of one or more external representations given how students interpret problems in engineering and the processes they use to select or construct solutions for these problems.