AI for Coding Education Meta-analyses: An Open-Science Approach that Combines Human and Machine Intelligence

Meta-analysis provides researchers with a way to assess the efficacy of an educational intervention across multiple independent studies by integrating them into a single statistical analysis, and thereby generalize over a larger, more heterogeneous population. This influences the ability to address goals of diversity, equity and inclusion (DEI), by providing a perspective over different populations of students. However, meta-analysis is extremely costly, mainly due to the need to manually code each of the many articles selected for inclusion, for each relevant variable. To shorten the time to publication, lower the cost, enhance transparency, and enable periodic updates of a given meta-analysis, we propose an open-science approach to meta-analysis coding that provides distinct modules for each variable, and that combines human and automated effort. We illustrate the approach on two variables that represent two types of automated support: pattern matching, versus machine learning. On the latter, we leverage a human-in-the loop approach for a variable that identifies distinct student populations, and is thus important for DEI: we report high accuracy of a neural model, and even higher accuracy of a selective prediction approach that defers to humans when the model output is insufficiently confident.