Using cognitive pretesting to explore causes for ethnic differences on role-plays

•The present study used cognitive pretesting to assess the causes of ethnic differences on role-plays.•Role-play texts were interpreted differently by ethnic majorities than ethnic minorities.•Ethnic differences in verbal ability explained only a part of the ethnic differences.•Evaluation of the texts indicated constructs that require specific cultural knowledge.

This study used cognitive pretesting to assess what factors could be causing score differences between ethnic majority and ethnic minority applicants on role-plays. In a laboratory room, equipped with a video camera, cognitive interviews were conducted among 12 ethnic majority and 12 ethnic minority students to examine ethnic differences in the interpretation of the instructional texts of two role-plays. Four trained assessors independently rated the participants’ videotaped responses. The assessors were asked to indicate whether the participants had understood the instructional text in general, the intended meaning of specific words or phrases, the problem, the job role, and the context of the situation. Ethnic differences were found for almost all of these aspects of the instructional texts (d-values varied between −0.02 and 0.89), indicating that the texts often did not activate the same concepts in the ethnic minority group as in the ethnic majority group. Ethnic differences in verbal ability explained only part of the ethnic differences in the interpretation of the instruction of the role-plays. Cognitive pretesting seems to be a valuable method to assess differences between ethnic majority and ethnic minority applicants in how they interpret test items and instructional texts.