Same text different processing? Exploring how raters’ cognitive and meta-cognitive strategies influence rating accuracy in essay scoring

• This paper links rater cognition with rating accuracy as estimated by MFRM statistics.
• Rating accuracy is operationalised as the match between raters’ ratings and expert ratings.
• Raters with different levels of accuracy show different patterns of cognitive and meta-cognitive behavior.
• More accurate raters are better at integrating information from target essays.
• More accurate raters are more self-conscious about their rating accuracy.

Rater variability has long been a concern in performance assessment. Among the various factors leading to variability, raters’ thinking processes may play an important role in shaping the rating outcomes. However, few studies have attempted to link raters’ cognition with rating accuracy. Thus, this study aims to explore how raters’ use of cognitive and meta-cognitive strategies might influence rating accuracy in the context of College English Test Band 4 essay scoring in China. Thirteen CET4 raters were classified into ACCURATE and LESS ACCURATE groups based on their rating accuracy, which was operationalised as the match between the ratings obtained from the target rater and those obtained from an expert panel measured by the Many-Facet Rasch Model analysis. Concurrent think-aloud was utilised to elicit the cognitive and meta-cognitive strategies the raters employed during the rating process. The thinking processes of the two rater groups were compared both quantitatively and qualitatively. The results suggest that more accurate raters are better at integrating information from target essays and are more self-conscious about their rating accuracy. Findings are expected to explain to some extent why some raters are effective in decision-making and generate more accurate ratings, which will have important implications on rater training.