Towards Human–Machine Collaboration in Creating an Evaluation Corpus for Adverse Drug Events in Discharge Summaries of Electronic Medical Records

Adverse drug events (ADEs) contribute significantly to morbidity and mortality in the healthcare system. The availability of digitalised hospitals' narrative clinical data offers a potentially rich resource to enhance pharmacovigilance efforts to manage potential safety issues arising from real-world use of drugs. The goal of this paper was to establish a foundation for creating an evaluation corpus by developing a set of annotation guidelines to achieve high inter-annotator agreement (IAA) and to evaluate the performance of basic entity identification tools for drugs, adverse events (AEs) and drug-AE relationships from 100 discharge summaries of a tertiary hospital in Singapore. Two teams of three annotators worked independently on text annotation using Knowtator. Three-way IAA of 86%, 70% and 49% were achieved for drugs, AEs and drug-AE relationships respectively. The performance of the machine algorithm was evaluated against annotations made by at least two annotators, with a recall of 84% and precision of 73% for drugs and a recall of 67% and precision of 53% for AEs. The high recall and precision for drug entity extraction suggests that machine pre-annotation of drugs followed by human annotation of AEs and drug-AE relationships could be a feasible approach in expediting the process of creating a larger evaluation corpus. Non-matches between machine and human annotations were examined to identify ways to further refine the algorithm. When successfully implemented, the identification of ADEs could greatly support pharmacovigilance work in characterising the magnitude and scope of ADEs and prioritising interventions to improve the drug safety.