Rater effects: Ego engagement in rater decision-making

The decision-making behaviors of 8 raters when scoring 39 persuasive and 39 narrative essays written by second language learners were examined, first using Rasch analysis and then, through think aloud protocols. Results based on Rasch analysis and think aloud protocols recorded by raters as they were scoring holistically and analytically suggested that rater background may have contributed to rater expectations that might explain individual differences in the application of the performance criteria of the rubrics when rating essays. The results further suggested that rater ego engagement with the text and/or author may have helped mitigate rater severity and that self-monitoring behaviors by raters may have had a similar mitigating effect.

► Rater ego-engagement with the text and/or author may have a mitigating effect on rater severity.
► Through self-monitoring with respect to the impact of construct-irrelevant variables raters may mitigate rater effects.
► Rater background contributes to rater expectations and influences scoring criteria used in rating writing samples.