Horseshoes, hand grenades, and treatment effects? Reassessing whether nonexperimental estimators are biased

•Tests bias in nonexperimental regression, propensity score approaches.•Rich pre-intervention data greatly reduce bias.•Some bias may remain.

Randomized controlled trials (RCTs) are considered the gold standard in estimating treatment effects. When an RCT is infeasible, regression modeling or statistical matching are often used instead. Nonexperimental methods such as these could produce unbiased estimates if the underlying assumptions hold, but those assumptions are usually not testable. Most prior studies testing nonexperimental designs find that they fail to produce unbiased estimates, but these studies have examined weaker evaluation designs. The present study addresses these limitations using student-level data based on a large-scale RCT of charter schools for which standardized achievement tests are the key outcome measure. The use of baseline data that are strongly predictive of the key outcome measures considerably reduces but might not completely eliminate bias.