Measuring rater judgment within learning assessments-Part 1: Why the number of categories matters in a rating scale

Assessments will focus learning, and so it should be aligned with desired educational outcomes. Frequently human raters (content experts) are needed to judge many abilities of advanced learners. To use subjective human judgments effectively, both the number of categories in a rating scale and the design of rubrics are important. This first article of the Methodology Matters section discusses cognitive limits in rater judgments, rating scales, and their applications. The second part of this article considers use of a “mixed approach” to rubric creation; holistic and analytic rubrics are described, as is dual-processing theory to help explain the advocated mixed approach. After reading part 1 of this article, readers should be able to (a) discuss why a four-point rating scale is often preferred (using cognitive psychology, avoiding a middle category, and rating scale performance from Rasch Measurement), and (b) create a preferred rating scale application for a learning assessment in pharmacy education. After reading part 2 of this article, the readers should be able to: (a) recognize the differences between holistic and analytic rubrics, (b) discuss integration with cognition and dual-processing theory, and (c) create a rubric for a learning assessment using a mixed approach.