Developing and validating measures for child welfare agencies to self-monitor fidelity to a child safety intervention

Building evidence of effective practice in child welfare requires practitioners and researchers to know the extent to which programs are implemented in order to understand evaluation results. Fidelity monitoring is a critical strategy for ensuring that evidence-based and promising practices are implemented as intended and can be studied in real-world contexts. This paper addresses challenges to measuring fidelity in child welfare systems and presents an approach taken with one state to define fidelity criteria and measure fidelity to a child safety intervention. Measurement challenges were addressed by using existing documents and case review mechanisms to assess fidelity, and measuring the quality of practitioner judgment using expert reviewers. Validity of fidelity criteria and fidelity review instruments was established through consensus with model developers and local purveyors. Twelve cases were reviewed by a panel of raters to assess inter-rater reliability and discrepancy between local purveyors and model developers. This participatory and capacity-building method can be replicated and used to develop and embed valid and reliable fidelity monitoring systems in public child welfare to continue to build evidence about what works in child welfare services.

► Fidelity is underutilized in child welfare evaluations.
► We develop criteria and test measures of fidelity to a child safety intervention.
► Consensus was used to establish content validity of fidelity measures.
► Measures demonstrate good reliability.
► Fidelity ratings of local purveyors are consistent with model developers.