TY - JOUR
T1 - Wise Crowd Content Assessment and Educational Rubrics
AU - Passonneau, Rebecca J.
AU - Poddar, Ananya
AU - Gite, Gaurav
AU - Krivokapic, Alisa
AU - Yang, Qian
AU - Perin, Dolores
N1 - Publisher Copyright:
© 2016, International Artificial Intelligence in Education Society.
PY - 2018/3/1
Y1 - 2018/3/1
N2 - Development of reliable rubrics for educational intervention studies that address reading and writing skills is labor-intensive, and could benefit from an automated approach. We compare a main ideas rubric used in a successful writing intervention study to a highly reliable wise-crowd content assessment method developed to evaluate machine-generated summaries. The ideas in the educational rubric were extracted from a source text that students were asked to summarize. The wise-crowd content assessment model is derived from summaries written by an independent group of proficient students who read the same source text, and followed the same instructions to write their summaries. The resulting content model includes a ranking over the derived content units. All main ideas in the rubric appear prominently in the wise-crowd content model. We present two methods that automate the content assessment. Scores based on the wise-crowd content assessment, both manual and automated, have high correlations with the main ideas rubric. The automated content assessment methods have several advantages over related methods, including high correlations with corresponding manual scores, a need for only half a dozen models instead of hundreds, and interpretable scores that independently assess content quality and coverage.
AB - Development of reliable rubrics for educational intervention studies that address reading and writing skills is labor-intensive, and could benefit from an automated approach. We compare a main ideas rubric used in a successful writing intervention study to a highly reliable wise-crowd content assessment method developed to evaluate machine-generated summaries. The ideas in the educational rubric were extracted from a source text that students were asked to summarize. The wise-crowd content assessment model is derived from summaries written by an independent group of proficient students who read the same source text, and followed the same instructions to write their summaries. The resulting content model includes a ranking over the derived content units. All main ideas in the rubric appear prominently in the wise-crowd content model. We present two methods that automate the content assessment. Scores based on the wise-crowd content assessment, both manual and automated, have high correlations with the main ideas rubric. The automated content assessment methods have several advantages over related methods, including high correlations with corresponding manual scores, a need for only half a dozen models instead of hundreds, and interpretable scores that independently assess content quality and coverage.
UR - http://www.scopus.com/inward/record.url?scp=85043253261&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85043253261&partnerID=8YFLogxK
U2 - 10.1007/s40593-016-0128-6
DO - 10.1007/s40593-016-0128-6
M3 - Article
AN - SCOPUS:85043253261
SN - 1560-4292
VL - 28
SP - 29
EP - 55
JO - International Journal of Artificial Intelligence in Education
JF - International Journal of Artificial Intelligence in Education
IS - 1
ER -