Honey, I shrunk the pooled SMD! Guide to critical appraisal of systematic reviews and meta-analyses using the Cochrane review on exercise for depression as example

• Evidence of exercise effects on depression is enveloped in confusion and controversy.
• Cochrane review on exercise for depression is extensively cited evidence synthesis.
• Scrutiny of methods reveals problems, including inclusion-exclusion criteria.
• Reanalysis with rational modifications shows large effect of exercise on depression.
• This analysis can serve as template for critical appraisal of other similar reviews.

ProblemIn several countries, physical activity is now recommended in clinical practice guidelines as an option for the treatment of subthreshold, mild, and moderate adult depression. However, most physicians do not present this option to their patients, attributing their decision to the perception that the supporting research evidence is inadequate. To assist readers in developing a strategy for evaluating pertinent research evidence, the present analysis offers a critical appraisal of the Cochrane systematic review and meta-analysis examining the effects of exercise on depression. Remarkably, successive updates of this review have reported a gradual “shrinkage” of the pooled standardized mean difference associated with exercise by 44%, from −1.10 in 2001 to −0.62 in 2013.MethodThe analysis evaluated the inclusion and exclusion criteria, the uniformity of rules, the rationale behind protocol changes, the procedures followed in assessing methodological quality, and reporting errors.ResultsInspection of the details of the review demystifies the “shrinkage” phenomenon, revealing that it is attributable to specific, questionable methodological choices and the fluidity of the review protocol. Reanalysis of the same database following rational modifications shows that the effect of exercise is large. Restricting the analysis to high-quality trials yields an effect size significantly different from zero.ConclusionsAlthough the clinical value of the Cochrane review is questionable, its educational potential is undeniable. Clinicians, students, referees, editors, systematic reviewers, guideline developers, and policymakers can use the present analysis as a template for evaluating the influence of methodological choices on the conclusions of systematic reviews and meta-analyses.