Evaluation of rule effectiveness and positive predictive value of clinical rules in a Dutch clinical decision support system in daily hospital pharmacy practice

IntroductionOur advanced clinical decision support (CDS) system, entitled ‘adverse drug event alerting system’ (ADEAS), is in daily use in our hospital pharmacy. It is used by hospital pharmacists to select patients at risk of possible adverse drug events (ADEs). The system retrieves data from several information systems, and uses clinical rules to select the patients at risk of ADEs. The clinical rules are all medication related and are formulated using seven risk categories.ObjectiveThis studies objectives are to 1) evaluate the use of the CDS system ADEAS in daily hospital pharmacy practice, and 2) assess the rule effectiveness and positive predictive value (PPV) of the clinical rules incorporated in the system.SettingLeiden University Medical Center, The Netherlands. All patients admitted on six different internal medicine and cardiology wards were included.MeasuresOutcome measures were total number of alerts, number of patients with alerts and the outcome of these alerts: whether the hospital pharmacist gave advice to prevent a possible ADE or not. Both overall rule effectiveness and PPV and rule effectiveness and PPV per clinical rule risk category were scored.Study designDuring a 5 month study period safety alerts were generated daily by means of ADEAS. All alerts were evaluated by a hospital pharmacist and if necessary, healthcare professionals were subsequently contacted and advice was given in order to prevent possible ADEs.ResultsDuring the study period ADEAS generated 2650 safety alerts in 931 patients. In 270 alerts (10%) the hospital pharmacist contacted the physician or nurse and in 204 (76%) cases this led to an advice to prevent a possible ADE. The remaining 2380 alerts (90%) were scored as non-relevant. Most alerts were generated with clinical rules linking pharmacy and laboratory data (1685 alerts). The overall rule effectiveness was 0.10 and the overall PPV was 0.08. Combination of rule effectiveness and PPV was highest for clinical rules based upon the risk category “basic computerized physician order entry (CPOE) medication safety alerts fine-tuned to high risk patients” (rule efficiency = 0.17; PPV = 0.14).ConclusionADEAS can effectively be used in daily hospital pharmacy practice to select patients at risk of potential ADEs, but to increase the benefits for routine patient care and to increase efficiency, both rule effectiveness and PPV for the clinical rules should be improved. Furthermore, clinical rules would have to be refined and restricted to those categories that are potentially most promising for clinical relevance, i.e. “clinical rules with a combination of pharmacy and laboratory data” and “clinical rules based upon the basic CPOE medication safety alerts fine-tuned to high risk patients”.