A scale for distinguishing efficacy from effectiveness was adapted and applied to stroke rehabilitation studies

ObjectivesTo apply a tool that purports to differentiate between efficacy and effectiveness studies to stroke rehabilitation trials and to evaluate its applicability and reliability.Study Design and SettingThree raters developed item operational definitions before independently applying the seven-item scale to 151 randomized controlled trials (RCT), published during or after 1997, that evaluated either a pharmacologic (P, n = 78) or a nonpharmacologic (NP, n = 73) intervention. Inter-rater reliability was assessed for both individual items and total scores, separately for P and NP trials.ResultsItem inter-rater reliability (multiple-rater kappa) ranged from 0.00 (95% CI [confidence interval]: −0.13, 0.13) to 0.85 (95% CI: 0.73, 0.98) and from 0.21 (95% CI: 0.08, 0.34) to 0.79 (95% CI: 0.66, 0.92) for P and NP RCTs, respectively. For the total score (dichotomized), kappa values were 0.43 (95% CI: 0.31, 0.56) and 0.51 (95% CI: 0.37, 0.64) for P and NP trials, respectively.ConclusionsThe tool provides a solid foundation on which to base further discussion of the differential criteria of efficacy–effectiveness trial design. Scale items should be properly operationalized depending on the research question of interest and evaluated for reliability before the scale is used for definitively judging a given study's design or the external validity of its results.