Learning When to Defer to Humans for Short Answer Grading

Zhaohui Li, Chengning Zhang, Yumi Jin, Xuesong Cang, Sadhana Puntambekar, Rebecca J. Passonneau

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations


To assess student knowledge, educators face a tradeoff between open-ended versus fixed-response questions. Open-ended questions are easier to formulate, and provide greater insight into student learning, but are burdensome. Machine learning methods that could reduce the assessment burden also have a cost, given that large datasets of reliably assessed examples (labeled data) are required for training and testing. We address the human costs of assessment and data labeling using selective prediction, where the output of a machine learned model is used when the model makes a confident decision, but otherwise the model defers to a human decision-maker. The goal is to defer less often while maintaining human assessment quality on the total output. We refer to the deferral criteria as a deferral policy, and we show it is possible to learn when to defer. We first trained an autograder on a combination of historical data and a small amount of newly labeled data, achieving moderate performance. We then used the autograder output as input to a logistic regression to learn when to defer. The learned logistic regression equation constitutes a deferral policy. Tests of the selective prediction method on a held out test set showed that human-level assessment quality can be achieved with a major reduction of human effort.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education - 24th International Conference, AIED 2023, Proceedings
EditorsNing Wang, Genaro Rebolledo-Mendez, Noboru Matsuda, Olga C. Santos, Vania Dimitrova
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages12
ISBN (Print)9783031362712
StatePublished - 2023
Event24th International Conference on Artificial Intelligence in Education, AIED 2023 - Tokyo, Japan
Duration: Jul 3 2023Jul 7 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13916 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference24th International Conference on Artificial Intelligence in Education, AIED 2023

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • General Computer Science

Cite this