External review letters in academic promotion and tenure decisions are reflective of reviewer characteristics

We examine validity and bias in external review letters (ERLs) in academic settings. ERLs play a critical role in promotion and tenure (P&T) decisions across the globe, ending careers in some cases while allowing other scientists' careers to flourish. We coded and analyzed 995 ERLs submitted by letter writers at various institutions as part of the P&T portfolios of 195 candidates at an R1 university. We examined their relationship with P&T committees' percent of positive votes by department, college, and university committees. We investigated how ERL linguistic features, letter writer characteristics, and candidate characteristics (productivity and gender) relate to P&T decision-making. Results show writer characteristics are more strongly related than candidate characteristics to ERL linguistic features associated with P&T decisions at the department and university levels. We develop recommendations for policymakers, including changes in the use of ERLs for the P&T process.