Observed score versus rasch score analysis of efficacy data: A case study comparison

Significant investments have been made by federal and private parties to examine the efficacy of promising educational interventions through large-scale efficacy trials in classroom settings. Many of these trials have yielded disappointing or inconclusive results. While there are innumerable forces that may be responsible for these ambiguous results (e.g., fidelity of implementation, lack of experimental or statistical control, intervening and unforeseen external intervention), we argue that perhaps the analytic approach used to examinethe data may be a large contributing factor. The dominant form of analysis requires the use of parametric statistics (What Works Clearinghouse, 2014). Parametric statistics assume a continuous measurement scale, an assumption that is not necessarily met when Classical Test Theory observed scores are used in education research. Rasch analysis may provide an approach for remediating the impact of violating the assumption of continuous measurement. To illustrate this analytic approach, we provide a case study demonstrating the results obtained from a science inquiry measure as analyzed using Classical Test Theory and Rasch analysis in a side-by-side comparison. The two different analytic approaches led to different conclusions about the efficacy of the program. While the Classical Test Theory observed scores indicated the program was not effective, the Rasch person measures indicated the opposite. The divergent outcomes show that the type of analysis used can potentially affect whether an intervention is found to be successful,. which may have implications for whether results are published and funding is continued.