Playing with the stakes: A consideration of an aspect of the social context of a gatekeeping writing assessment

In high-stakes writing assessments, rater training in the use of a rating scale does not eliminate variability in grade attribution. This realisation has been accompanied by research that explores possible sources of rater variability, such as rater background or rating scale type. However, there has been little consideration thus far of variability in rater behaviour that may be brought about by the socio-political context of the testing situation. In addition, studies of rater behaviour are undertaken in either research conditions or authentic high-stakes testing conditions, but never both. This report is a “stakes study” in the context of a writing test for teacher certification in Quebec. The research objective was to see if raters grade differently when the consequences to test takers are changed but if all else remains the same (text, rater, rating scale and rater training). Scores given by the raters in the “high-stakes” (authentic testing) and “low-stakes” (research) conditions were compared. Raters were also questioned immediately after their final grading session to allow them to discuss any perceived differences in their rating behaviour in the two conditions. It was hypothesised that raters might be more severe in their grading if there were no stakes involved for the test takers. This case study, while exploratory and small in scale, demonstrates the potential in including the social context as a factor in future research on variability in rater scoring behaviour.