Predictive models of safety based on audit findings: Part 1: Model development and reliability

This consecutive study was aimed at the quantitative validation of safety audit tools as predictors of safety performance, as we were unable to find prior studies that tested audit validity against safety outcomes. An aviation maintenance domain was chosen for this work as both audits and safety outcomes are currently prescribed and regulated. In Part 1, we developed a Human Factors/Ergonomics classification framework based on HFACS model (Shappell and Wiegmann, 2001a and Shappell and Wiegmann, 2001b), for the human errors detected by audits, because merely counting audit findings did not predict future safety. The framework was tested for measurement reliability using four participants, two of whom classified errors on 1238 audit reports. Kappa values leveled out after about 200 audits at between 0.5 and 0.8 for different tiers of errors categories. This showed sufficient reliability to proceed with prediction validity testing in Part 2.

► We developed a Human Factors classification framework to analyze audit records.
► It was tested for reliability with Kappa values at 0.5–0.8 for different error classes.
► This result showed sufficient reliability to proceed with predictive validity testing.