Jackknife evaluation of uncertainty judgments aggregated by the Kullback–Leibler distance

One feasible approach to aggregating uncertainty judgments in risk assessments is to use calibration variables (or seed questions) and the Kullback–Leibler (K–L) distance to evaluate experts’ substantive or normative expertise and assign weights based on the corresponding scores. However, the reliability of this aggregation model and the effects of the number of seed questions or experts on the stability of the aggregated results are still at issue. To assess the stability of the aggregation model, this study applies the jackknife re-sampling technique to a large data set of real-world expert opinions. We also use a nonlinear regression model to analyze and interpret the resulting jackknife estimates. Our statistical model indicates that the stability of Cooke’s classical model, in which the components of the scoring rule are determined by the K–L distance, increases exponentially as the number of seed questions increases. Considering the difficulty and importance of creating and choosing appropriate seed variables, the results of this study justify the use of the K–L distance to determine and aggregate better probability interval or distribution estimates.