Teachers' rankings of children's executive functions: Validating a methodology for school-based data collection

We developed a novel, vignette-based ranking procedure to simultaneously collect teacher-reported executive function (EF) data for all students in a classroom. This ranking measure is an improvement over existing Likert-type rating scales because it can be completed more quickly and with comparatively little effort by teachers. Data for this validation study were drawn from a large, school-based study of third, fourth, and fifth graders (N = 813 from 33 classrooms in eight schools) in which ranking data and direct assessments of EF were collected. Using a subsample of students for whom teachers' ratings of EF and school records data were also collected (N = 311), we demonstrated that teachers' rankings of EF showed high convergent validity with teachers' ratings of EF and that both teacher-reported measures showed similar convergent validity with direct assessments of EF and similar predictive validity with respect to students' scores on standardized English/language arts and math achievement tests. Using data from the larger sample (N = 813), we conducted a simulation study demonstrating that the impact of missing data on the association between the rankings and the direct assessments of EF is minimal. Based on these results, the ranking procedure is a methodological innovation that enables the collection of relatively high-quality teacher-reported EF data for all students in a classroom quickly and with minimal burden on teachers. This vignette-based assessment method could be adapted to other domains of non-academic skills. We discuss varied uses of the ranking method for researchers and practitioners.