The role of treatment acceptability, effectiveness, and understanding in treatment fidelity: Predicting implementation variation in a middle school science program

•Teacher variation in implementing school-based programs complicates evaluation.•Implementation may be affected by acceptability, effectiveness, and understanding.•Newly developed assessments gauge these constructs for school-based intervention.•Assessments showed modest associations with implementation of the program.•Measures show promise for evaluation and research on implementation fidelity.

Evaluating a school-based program is a particular challenge when teachers implementing the program. Variations in implementation can be difficult to measure and predict. We developed measures to explore variation in treatment implementation and serve as predictors of variation in a school-based science education program. Based on previous work, we focused on assessing treatment acceptability, effectiveness, and understanding among teachers as critical determinants of variations in program implementation using self-reported and objective measures of implementation. All three constructs were associated with implementation of the program. Our measures of these constructs show promise for use in formative and summative evaluations. Our stratification of program elements with implementation predictors can provide a template for future exploration of treatment fidelity.