Evaluating systems change in a juvenile justice reform initiative

Evaluating comprehensive, interagency initiatives to reform human services systems presents substantial challenges to traditional research models. Outcomes are observed at the system level rather than the individual level, and the validity of study results may be challenged on a variety of dimensions, particularly small sample sizes and measurement error. We report the results of a cross-site evaluation of the first phase of Reclaiming Futures, a five-year effort to improve services and interventions for justice-involved youth in 10 communities across the United States. We present the evaluation findings and discuss implications for similar research endeavors. The methods used in this study may be useful for other system-level evaluation efforts. A social network analysis analyzes changes in the size, density, and cohesiveness of stakeholder networks. Bivariate models test for associations between Reclaiming Futures implementation and key stakeholder perceptions of effective system change. Multivariate models are used to explore selection effects on item response. In general, we find positive correlations between successful implementation of Reclaiming Futures and indicators of effective and efficient system change.

Research Highlights
► We find the 10 communities exhibited fidelity to the Reclaiming Futures model.
► Social network analysis showed increased strength of agency networks.
► Twelve of 13 system performance indices had statistically significant improvements.
► Largest gains in treatment efficacy, systems integration, screening and assessment.