AI for Coding Education Meta-analyses: An Open-Science Approach that Combines Human and Machine Intelligence

Vipul Gupta, Brian R. Belland, Alexander Billups, Rebecca J. Passonneau

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

Meta-analysis provides researchers with a way to assess the efficacy of an educational intervention across multiple independent studies by integrating them into a single statistical analysis, and thereby generalize over a larger, more heterogeneous population. This influences the ability to address goals of diversity, equity and inclusion (DEI), by providing a perspective over different populations of students. However, meta-analysis is extremely costly, mainly due to the need to manually code each of the many articles selected for inclusion, for each relevant variable. To shorten the time to publication, lower the cost, enhance transparency, and enable periodic updates of a given meta-analysis, we propose an open-science approach to meta-analysis coding that provides distinct modules for each variable, and that combines human and automated effort. We illustrate the approach on two variables that represent two types of automated support: pattern matching, versus machine learning. On the latter, we leverage a human-in-the loop approach for a variable that identifies distinct student populations, and is thus important for DEI: we report high accuracy of a neural model, and even higher accuracy of a selective prediction approach that defers to humans when the model output is insufficiently confident.

Original languageEnglish (US)
Title of host publicationLecture Notes on Data Engineering and Communications Technologies
PublisherSpringer Science and Business Media Deutschland GmbH
Pages14-29
Number of pages16
DOIs
StatePublished - 2023

Publication series

NameLecture Notes on Data Engineering and Communications Technologies
Volume190
ISSN (Print)2367-4512
ISSN (Electronic)2367-4520

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Media Technology
  • Computer Science Applications
  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Cite this