TY - JOUR
T1 - Taking PISA Seriously
T2 - How Accurate are Low-Stakes Exams?
AU - Akyol, Pelin
AU - Krishna, Kala
AU - Wang, Jinwen
N1 - Funding Information:
We are grateful to participants at the Econometrics Society World Congress in 2020, Econometric Society meetings in Shanghai, China in 2018, International Association of Applied Econometrics Conference in Cyprus in 2019, Conference of the European Society for Population Economics (ESPE) in Bath, UK in 2019 and 9th ifo Dresden Workshop on Labor Economics and Social Policy in 2019. We would particularly like to thank Joris Pinkse, Keisuke Hirano, and Kim Ruhl for their comments and suggestions and Meghna Bramhachari for help in proofreading. We owe special thanks to colleagues at the OECD for answering our numerous questions about the data. Huacong Liu was instrumental in our working on this project, and we thank her for all her help. We are responsible for all errors.
Publisher Copyright:
© 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC part of Springer Nature.
PY - 2021/6
Y1 - 2021/6
N2 - PISA is seen as the gold standard for evaluating educational outcomes worldwide. Yet, being a low-stakes exam, students may not take it seriously resulting in downward biased scores and inaccurate rankings. This paper provides a method to identify and account for non-serious behavior in low-stakes exams by leveraging information in computer-based assessments in PISA 2015. Our method corrects for non-serious behavior by fully imputing scores for items not taken seriously. We compare the scores/rankings calculated by our method to the scores/rankings calculated by giving zero points to skipped items as well as to the scores/rankings calculated by treating skipped items at the end of the exam as if they were not administered, which is the procedure followed by PISA. We show that a country can improve its ranking by up to 15 places by encouraging its own students to take the exam seriously and that the PISA approach corrects for only about half of the bias generated by the non-seriousness.
AB - PISA is seen as the gold standard for evaluating educational outcomes worldwide. Yet, being a low-stakes exam, students may not take it seriously resulting in downward biased scores and inaccurate rankings. This paper provides a method to identify and account for non-serious behavior in low-stakes exams by leveraging information in computer-based assessments in PISA 2015. Our method corrects for non-serious behavior by fully imputing scores for items not taken seriously. We compare the scores/rankings calculated by our method to the scores/rankings calculated by giving zero points to skipped items as well as to the scores/rankings calculated by treating skipped items at the end of the exam as if they were not administered, which is the procedure followed by PISA. We show that a country can improve its ranking by up to 15 places by encouraging its own students to take the exam seriously and that the PISA approach corrects for only about half of the bias generated by the non-seriousness.
UR - http://www.scopus.com/inward/record.url?scp=85103214610&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85103214610&partnerID=8YFLogxK
U2 - 10.1007/s12122-021-09317-8
DO - 10.1007/s12122-021-09317-8
M3 - Article
AN - SCOPUS:85103214610
SN - 0195-3613
VL - 42
SP - 184
EP - 243
JO - Journal of Labor Research
JF - Journal of Labor Research
IS - 2
ER -