The effects of accent familiarity and language teaching experience on raters' judgments of non-native speech

Rater effects, defined as the construct-irrelevant variation associated with rater characteristics, are critical to the reliability and validity of speaking assessments ( Bachman et al., 1995; Kunnan, 2000). The current study investigates two under-explored characteristics among untrained raters: familiarity with the speakers' non-native accents and experience in teaching English as a second/foreign language (ESL/EFL). This study is the first to use a novel, objective measure of accent familiarity, i.e., an accent identification task. It included three groups of raters (n = 22 in each group) who varied on the two characteristics. All raters assigned holistic and three analytical ratings (foreign accents, grammar and vocabulary, and content) to 26 speech files. They also filled out a survey reporting their demographic information, perceived impacts of their background on their ratings, and evaluative features they used for rating.Although there were no significant differences in the three groups' ratings, ESL/EFL teachers self-reported that their background affected their rating decisions, and one third of them also felt that they were more lenient. Compared to non-teachers, ESL/EFL teachers were also better able to separate the analytical dimensions and were less biased by speakers' foreign accents when judging the overall proficiency or the content of their speech.