Teacher quality differences between teacher preparation programs: How big? How reliable? Which programs are different?

• 16 states hold teacher preparation programs (TPPs) accountable for teacher quality.
• Yet it is hard to single out TPPs whose teachers are exceptionally good or bad.
• In Texas, true teacher quality differences between TPPs are small, and estimates of those differences are noisy.
• Running multiple tests on noisy estimates risks misclassifying TPPs as good or bad.
• TPP accountability efforts may lead to arbitrary and ineffective policy decisions.

Sixteen US states have begun to hold teacher preparation programs (TPPs) accountable for teacher quality, where quality is estimated by teacher value-added to student test scores. Yet it is not easy to identify TPPs whose teachers are substantially better or worse than average. True teacher quality differences between TPPs are small; estimated differences are not very reliable; and when many TPPs are compared, multiple comparisons increase the danger of misclassifying ordinary TPPs as good or bad. Using a large and diverse dataset from Texas, we evaluate statistical methods for estimating teacher quality differences between TPPs. The most convincing estimates come from a value-added model where confidence intervals are widened by the inclusion of teacher random effects (or teacher clustering in large TPPs) and further widened by the Bonferroni correction for multiple comparisons. Using these confidence intervals, it is rarely possible to tell which TPPs, if any, are better or worse than average. The potential benefits of TPP accountability may be too small to balance the risk that a proliferation of noisy TPP estimates will encourage arbitrary and ineffective policy actions.