There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review

ObjectivesTo assess the consistency in risk of bias (RoB) judgments across Cochrane reviews for studies appearing in more than one Cochrane review in the field of subfertility.Study Design and SettingWe retrieved any study that had been used more than once in systematic reviews present on the Cochrane Database of Systematic Reviews in the area of subfertility. We then retrieved the recorded RoB assessments for these studies and looked at the consistency of judgments made between different authoring teams on the same trials.ResultsFrom the 156 bias judgments that were completed by at least two separate groups of authors, 45% of these judgments differed. For the domains of random sequence generation and incomplete outcome data, there was reasonably high level of agreement (71% and 79%, respectively). However, for the domain of blinding, agreement was reached in only 35% of cases.ConclusionThis assessment of how consistently the RoB is being applied in Cochrane reviews has shown that, especially in some domains, there are large discrepancies in how RoB is being evaluated. Further work needs to be undertaken to improve the application of this tool.