Methods for systematic reviews of administrative database studies capturing health outcomes of interest

• Few studies using administrative databases reported validation of case-finding algorithms.
• Studies reporting validation typically calculated positive predictive value.
• The applicability of algorithms identified should be carefully assessed prior to use.

This report provides an overview of methods used to conduct systematic reviews for the US Food and Drug Administration (FDA) Mini-Sentinel project, which is designed to inform the development of safety monitoring tools for FDA-regulated products including vaccines. The objective of these reviews was to summarize the literature describing algorithms (e.g., diagnosis or procedure codes) to identify health outcomes in administrative and claims data. A particular focus was the validity of the algorithms when compared to reference standards such as diagnoses in medical records. The overarching goal was to identify algorithms that can accurately identify the health outcomes for safety surveillance. We searched the MEDLINE database via PubMed and required dual review of full text articles and of data extracted from studies. We also extracted data on each study's methods for case validation. We reviewed over 5600 abstracts/full text studies across 15 health outcomes of interest. Nearly 260 studies met our initial criteria (conducted in the US or Canada, used an administrative database, reported case-finding algorithm). Few studies (N = 45), however, reported validation of case-finding algorithms (sensitivity, specificity, positive or negative predictive value). Among these, the most common approach to validation was to calculate positive predictive values, based on a review of medical records as the reference standard. Of the studies reporting validation, the ease with which a given clinical condition could be identified in administrative records varied substantially, both by the clinical condition and by other factors such as the clinical setting, which relates to the disease prevalence.