The assessment of mathematical literacy of linguistic minority students: Results of a multi-method investigation

• This study investigates the thinking of students living in minority settings while taking PISA items that are biased against this population.
• Four levels of quantitative and qualitative investigation were conducted.
• Experts identified many possible linguistic aspects that might bias test takers.
• Students manage to do and succeed on PISA items even when they experienced language problems.

Assessing mathematical literacy of students who have limited proficiency in the language of the test is a critical challenge in mathematics education. Previous research indicates that knowledge and competencies of such students are underestimated. This presents a major validity and fairness problem for assessment. Most efforts addressing fairness and validity issues in assessment of linguistic minority students focus on the test language only. To overcome limitations of single approaches, we examine in this study the interaction between the test language and the student language background by means of multiple methods. Thus, we investigate possible linguistic bias of items flagged as functioning differentially (the result of DIF analyses) by means of (a) two levels of expert analyses and (b) student think-aloud protocols to investigate language effects in published mathematics items from the 2000 to 2003 Programme for International Student Assessment (PISA) administration for students attending French schools in Canada and speaking either French or other languages at home. DIF analyses were conducted to identify items on which students from different home language backgrounds attending French schools achieve differently. The expert panels tended to identify surface characteristics of language that may be responsible for group differences but not for the differential effects detected by differential item functioning (DIF). Student think-aloud protocols in part confirm and in part contradict DIF results, providing insights for the source of the differences. Suggestions are provided for further study.