A cautionary tale on using panel data estimators to measure program impacts

- We compare experimental and observational estimates of environmental program impact.
- We expand the sample of comparison units to improve covariate balance.
- Despite similarity of covariates and baseline trends, bias of the estimator worsens.
- Fixed-effects panel estimators and indirect tests of their validity are no panacea.

We compare experimental and nonexperimental estimates from a social and informational messaging experiment. Our results show that applying a fixed effects estimator in conjunction with matching to pre-process nonexperimental comparison groups cannot replicate an experimental benchmark, despite parallel pre-intervention trends and good covariate balance. The results are a stark reminder about the role of untestable assumptions-in our case, conditional bias stability-in drawing causal inferences from observational data, and the dangers of relying on single studies to justify program scaling-up or canceling.