Adapting CEF-descriptors for rating purposes: Validation by a combined rater training and scale revision approach

We explore how a local rating scale can be based on the Common European Framework CEF-proficiency scales. As part of the scale validation (Alderson, 1991 and Lumley, 2002), we examine which adaptations are needed to turn CEF-proficiency descriptors into a rating scale for a local context, and to establish a practicable method to revise the initial scale-draft while simultaneously training raters for trialling the rating scale. We propose a data-driven approach focusing on the descriptors, involving raters in the scale revision as part of the scale validation process (Barkaoui, 2010a and Cohen, 1994). Rating data were analysed during a two-month period, employing 19 writing tasks and 13 raters; individual descriptors and raters were monitored for consistency. Our approach facilitated identifying causes of low agreement, let it be raters or scale wording, which in turn allowed targeted feedback to raters and revisions of scale-wording where indicated. We conclude with a summary of necessary adaptations of CEF-descriptors for rating purposes, and we outline further steps to validate the revised rating scale. We aim to contribute to improve rater training and rating scale development based on the CEF, since the literature for ‘real’ test development contexts seems rather scarce (Barkaoui, 2010b, Knoch, 2011a and Knoch, 2011b).

► We demonstrate how CEF-descriptors can be adapted for a local rating context.
► Rating data from 19 tasks and 13 raters were analysed over 2 months.
► Monitoring agreement for descriptors allows identifying causes for inconsistencies.
► Our approach permits effective rater feedback and revision of descriptor wording.
► Involving raters in scale revision leads to more consistent rating.