Evaluating the effectiveness of actuarial risk assessment models

A series of studies provides evaluative evidence on actuarial models of risk assessment in child welfare using both scientific and practical criteria. Two true experiments with random assignment explore the effects of relatively successful and unsuccessful models. Results of the first indicate that valid models influenced caseworker risk estimates, yet the models were not superior to the judgments made by caseworkers not exposed to the models. In the second, three failed models and one successful model influenced caseworker risk estimates, yet even the one successful model was not found to be superior to caseworker judgment. The relationship between model predictability, influence, comprehension and efficiency is underscored in these studies. The final study provided some caseworkers with feedback from an actuarial model in the form of an “alert” that required them to take action on an actual case in the field. Some workers were not exposed to the feedback. Results indicated that although the alert status of a case affected caseworks judgments and case-related behavior in expected directions, the effect of knowledge of the actuarial feedback on actual case-related behavior was small.