Validating THERP: Assessing the scope of a full-scale validation of the Technique for Human Error Rate Prediction

Science-based Human Reliability Analysis (HRA) seeks to experimentally validate HRA methods in simulator studies. Emphasis is on validating the internal components of the HRA method, rather than the validity and consistency of the final results of the method. In this paper, we assess the requirements for a simulator study validation of the Technique for Human Error Rate Prediction (THERP), a foundational HRA method. The aspects requiring validation include the tables of Human Error Probabilities (HEPs), the treatment of stress, and the treatment of dependence between tasks. We estimate the sample size, n, required to obtain statistically significant error rates for validating HEP values, and the number of observations, m, that constitute one observed error rate for each HEP value. We develop two methods for estimating the mean error rate using few observations. The first method uses the median error rate, and the second method is a Bayesian estimator of the error rate based on the observed errors and the number of observations. Both methods are tested using computer-generated data. We also conduct a pilot experiment in The Ohio State University's Nuclear Power Plant Simulator Facility. Student operators perform a maintenance task in a BWR simulator. Errors are recorded, and error rates are compared to the THERP-predicted error rates. While the observed error rates are generally consistent with the THERP HEPs, further study is needed to provide confidence in these results as the pilot study sample size is small. Sample size calculations indicate that a full-scope THERP validation study would be a substantial but potentially feasible undertaking; 40 h of observation would provide sufficient data for a preliminary study, and observing 101 operators for 20 h each would provide data for a full validation experiment.