Research papersTreatment of multiple test readers in diagnostic accuracy systematic reviews-meta-analyses of imaging studies

•Imaging systematic reviews rarely report how they handle multiple reader data.•Most imaging systematic reviews include studies with more than one reader.•Imaging systematic reviews use different strategies for handling multiple reader data.•Optimal method for handling multiple reader data imaging reviews is presently unknown.•Development of a two-level hierarchical model may be necessary to address this.

ObjectiveTo evaluate the handling of multiple readers in imaging diagnostic accuracy systematic reviews-meta-analyses.MethodsSearch was performed for imaging diagnostic accuracy systematic reviews that performed meta-analysis from 2005-2015. Handling of multiple readers was classified as: 1) averaged; 2) 'best' reader; 3) 'most experienced' reader; 4) each reader counted individually; 5) random; 6) other; 7) not specified. Incidence and reporting of multiple reader data was assessed in primary diagnostic accuracy studies that were included in a random sample of reviews.ResultsOnly 28/296 (9.5%) meta-analyses specified how multiple readers were handled: 7/28 averaged results, 2/28 included the best reader, 14/28 treated each reader as a separate data set, 1/28 randomly selected a reader, 4/28 used other methods. Sample of 27/268 'not specified' reviews generated 442 primary studies. 270/442 (61%) primary studies had multiple readers: 164/442 (37%) reported consensus reading, 87/442 (20%) reported inter-observer variability, 9/442 (2%) reported independent datasets for each reader. 26/27 (96%) meta-analyses contained at least one primary study with multiple readers.ConclusionsReporting how multiple readers were treated in imaging systematic reviews-meta-analyses is uncommon and method used varied widely. This may result from a lack of guidance, unavailability of appropriate statistical methods for handling multiple readers in meta-analysis, and sub-optimal primary study reporting.