Rater calibration when observational assessment occurs at large scale: Degree of calibration and characteristics of raters associated with calibration

Observational assessment is used to study program and teacher effectiveness across large numbers of classrooms, but training a workforce of raters that can assign reliable scores when observations are used in large-scale contexts can be challenging and expensive. Limited data are available to speak to the feasibility of training large numbers of raters to calibrate to an observation tool, or the characteristics of raters associated with calibration. This study reports on the success of rater calibration across 2093 raters trained by the Office of Head Start (OHS) in 2008–2009 on the Classroom Assessment Scoring System (CLASS), and for a subsample of 704 raters, characteristics that predict their calibration. Findings indicate that it is possible to train large numbers of raters to calibrate to an observation tool, and rater beliefs about teachers and children predicted the degree of calibration. Implications for large-scale observational assessments are discussed.

► Observation is being used to assess program and teacher quality at large scale.
► Data are needed on feasibility of training large numbers of observers.
► Efforts by Office of Head Start demonstrate large-scale rater training possible.
► Rater characteristics associated with calibration are explored.