Pair assessment of pupil writing: A dialogic approach for studying the development of rater competence

•Novice raters within an assessment panel were studied over a period of 10 months.•The raters assessed pupil texts in pairs and groups.•Stable use of referents is indicative of small changes in rater behaviour.•This is problematic in a community of practice where learning/changes are intended.•More focus on use of common standards within the panel seems to be needed.

This paper reports on rating during the development of a Norwegian sample-based national assessment of L1 writing as a key competency. This assessment is to be officially introduced in August 2014. Novice members of a national rater panel to assess Year 8 pupils’ texts were studied during three of their successive training sessions: in June 2011, November 2011 and April 2012. My purpose was to conduct an exploratory investigation into how the rating practice of novice raters might develop during such a preparatory stage. The raters in this study mainly assessed in pairs, and data sources were assessment dialogues. The analysis of transcripts showed that rater behaviour changed only to a minor extent towards an increased use of shared assessment resources. The quality of the assessment dialogues did not change much either, leaving the impression that raters often reached consensus without much discussion. Since reliable scoring is a collective task, however, it is argued that a more balanced use of resources, both those attained from teacher practice and those attained from being a member of a national panel, may together with more exploration-oriented dialogue be necessary to achieve sufficient reliability.