The matter of assessor variance in early childhood education—Or whose score is it anyway?

Useful assessment outcomes (as manifest through assigned scores) must show reasonable variation across children because it is that variation that presumably defines children's individual differences. Alternatively it is conceivable that some portion of the variability in assessment outcomes does not reflect child differences but rather differences in the performance of the assessors who carry out assessments. Hierarchical linear modeling is applied in this article to identify the amount of score variation attributable to assessors rather than children. Working with multiple cohorts of Head Start and kindergarten children, score variation is analyzed for measures administered outside of classrooms by extramural assessors and for teacher-administered measures within classrooms. The amount of assessor variance (vs. actual child variance) was negligible as associated with extramural assessors but substantial for teacher assessors, indicating that large portions of the variability in teacher-administered assessments have nothing to do with children's unique performances. Recommendations are provided to assist the interpretation of assessment outcomes in future research and practice.

• Variability in children's scores is attributable to assessors as well as the children themselves. • Assessments administered by teachers show substantially more assessor variance. • Extramural assessors produce the least amount of assessor variance.