Nonrandomized quality improvement intervention trials might overstate the strength of causal inference of their findings

ObjectiveTo assess the strength of causal inferences reported in randomized and nonrandomized evaluations of quality improvement (QI) interventions in relation to the study design and the direction of results for the primary outcomes.Study Design and SettingWe searched 11 journals for QI intervention studies that aimed to change clinician behavior. Statements that addressed the causal inference between intervention and outcomes were extracted and were rated by 34 researchers for the strength of causality.ResultsWe found 38 randomized controlled trials (RCTs) and 35 non-RCTs, and extracted 68 quotes from the abstracts and 139 from the main text. A significant interaction was found between study design and direction of results for the abstract quotes (P = 0.022). The ratings for non-RCTs were higher when the results were mixed, but for RCTs, they were higher if the results were positive or no effect, although none of the differences were statistically significant at α = 0.05 after adjusting for multiple comparisons. For the main text quotes, the causality rating was higher by 0.43 for RCTs than for non-RCTs after adjusting for the direction of results (P < 0.001).ConclusionAuthors might have overstated the strength of causal inference in the abstracts of non-RCTs, but appeared to report causality appropriately in the main text.