An investigation of dependence in expert judgement studies with multiple experts

Expert judgement plays an important role in forecasting and elsewhere, as it can be used to quantify models when no data are available and to improve predictions from models when combined with data. In order to provide defensible estimates of unknowns in an analysis, the judgements of multiple experts can be elicited. Mathematical aggregation methods can then be used to combine these individual judgements into a single judgement for the decision maker. However, most mathematical aggregation methods assume that such judgements come from experts who are independent, which is unlikely to be the case in practice. This paper investigates dependence in expert judgement studies, both within and between experts. It provides the most comprehensive analysis to date by considering all studies in the TU Delft database. It then assesses the practical significance of the dependencies identified in the studies by comparing the performances of several mathematical aggregation methods with varying dependence assumptions. Between-expert correlations were more prevalent than within-expert correlations. For studies that contained between-expert correlations, models which include these produced better forecasts. The implications of this for the use of expert judgement in forecasting are discussed.