Inter-rater reliability and false confidence in precision: Using standard error of measurement within PharmD admissions essay rubric development

The Accreditation Council for Pharmacy Education requires that written communication be assessed in Doctor of Pharmacy admissions processes. Reliability is a standard for ethical testing, and inter-rater reliability with scoring essays necessitates continued quality assurance. Both inter-rater consistency and inter-rater agreement are part of inter-rater reliability and so both need scrutiny. Within our admission process, we analyzed inter-rater reliability for faculty rater essay scores from 2008–2012 using intraclass correlation (ICC) for consistency and standard error of measurement (SEM) for agreement. Trends in these scores were examined to evaluate the impact of rubric implementation, revisions, and rater training integrated over the course of those five admission cycles. For regular admission (RA) candidates, an analytic rubric was implemented in 2009. Scoring without a rubric began with an ICC of 0.595 (2008) and improved to 0.860 (2012) after rubric implementation, revisions, and rater training. In a separate but similar process for contingent admission (CA) candidates, a holistic rubric was implemented in 2010. The ICC for CA essay scoring before rubric was 0.586 (2009), and it improved to 0.772 (2012). With both rubrics, inter-rater agreement (using SEM) improved with smaller scoring scales (i.e., 4-point > 20-point > 50-point). In our experience, rubric implementation and training appeared to improve inter-rater consistency, though inter-rater agreement was not improved with every rubric revision. Our holistic rubrics’ 4-point scale was most precise for both inter-rater consistency and inter-rater agreement. Our rubrics with larger scoring scales appeared to foster false confidence in precision of scores—with larger variation in scores introducing more measurement error.