The decline effect in evaluations of the impact of the Strengthening Families Program for Youth 10-14 (SFP 10-14) on adolescent substance use

The decline effect refers to the situation whereby an initially positive set of research findings concerning a phenomenon fail to be replicated in subsequent studies. One explanation for the occurrence of the decline effect is that the initial positive results reported in publications from a single research team are the product of the use of flexible data analysis practices and selective reporting. The present paper reviews evaluations of the Strengthening Families Program for Youth 10-14 (SFP 10-14) conducted by the research group who developed the program and five independent replication studies. The focus is on the effects reported for adolescent substance use. It is argued that the isolated statistically significant results reported in the original studies were produced through the use of flexible data analysis practices such as changing outcome variables across publications and post hoc alterations in study design. In light of this, it is unsurprising that the five independent evaluations have failed to replicate these results. A strength of three of the independent evaluations is that they were registered in a clinical trials registry and had published their study protocols. Both of these procedures limit the opportunity to conduct flexible data analysis. It is recommended that future evaluations of the SFP 10-14 should employ these procedures as well as utilizing the registered reports publication process, making their data and methods openly available to other researchers, and including a skeptic in their research team.