Adjusting weighted kappa for severity of mistriage decreases reported reliability of emergency department triage systems: a comparative study

ObjectiveMistriage can have serious consequences for patients with urgent complaints. We therefore reviewed the assessment of triage-reliability, and propose an alternative weighted kappa that accounts for severity of mistriage.Study Design and SettingA systematic literature search was performed in three online databases and a search engine. An alternative kappa weighting scheme was developed (triage-weighted kappa), and kappas of previous conducted reliability studies were recalculated.ResultsKappa is the most frequently used statistic in triage-reliability studies (n = 33). More than half of the studies did not report the type of kappa that was used. Linear and quadratically weighted kappa values do not reflect the seriousness of mistriage. Several studies reported almost perfect agreement, whereas percentages of mistriage ranged between 11.1% and 43.4%. In all studies, triage-weighted kappa was lower than reported kappas with a mean difference of 0.17 (range: 0.04–0.32).ConclusionNo existing studies on reliability of triage systems account for mistriage. Using triage-weighted kappa, which reflects severity of mistriage, shows that the reliability of triage systems is lower than reported.