Evaluating program effectiveness: A case study of the School-to-Work Opportunities Act in California

Evaluations of the effectiveness of government programs may serve both to improve program implementation and to convey to policymakers information on the effectiveness of these programs. However, the evidence generated from programs and their providers—perhaps especially when these programs are decentralized and implemented and run locally—may fall far short of standards for convincing evidence, raising questions about the decisionmaking of local providers based on this evidence, and the quality of the evidence provided to policymakers.This paper presents a case study of this question in the context of the federal School-to-Work Opportunities Act in California, which was implemented by decentralized Local Partnerships (LPs) throughout the state. Extensive interviews with representatives of LPs in California and a detailed review of local evaluations are used to study how school-to-work providers gauge the success of their programs and to assess evaluations of these programs carried out or commissioned by these LPs. The findings raise serious doubts about the adequacy of evaluation efforts by local program providers, and the case study provides little or no convincing evidence of the effectiveness of these programs.