Improving the Delphi process: Lessons from social psychological research

The Delphi technique was largely developed to avoid the problems of freely interacting groups such as dominant individuals and pressure to conform to the majority view. Our review of the social psychological literature reveals some obstacles to Delphi achieving its full potential relative to other cheaper and easier methods of aggregating judgment. We identify residual normative and informational pressures towards consensus that potentially reduce process gain that might otherwise be achieved. For instance, panelist confidence may act as a signal of status rather than be a valid cue to expertise, whereas consensus appears to have a strong influence on the final outcome that can reduce its accuracy when there are valid minority opinions. We argue that process gain in Delphi must occur through those further from the "truth" changing their opinion more than those closer to the truth, with the general direction of opinion change being towards the truth. For such virtuous opinion change to occur we suggest the need to both facilitate opinion change and provide good cues as to where the truth lies. Research on Judge Advisor Systems shows that people usually do not change their opinion as much as they should, giving too much weight to their own opinion and too little to the views of others—this bias can be reduced by increasing involvement and motivation. In addition, we propose that the best way to provide good cues as to the direction of the truth is to elicit rich reasoning from panelists about the judgment or choice in question, then use this as feedback. We suggest practical ways of focusing and deepening panelists’ consideration and evaluation of such reasoning—such that all proffered opinions are well-evaluated. Additionally, we propose a model of opinion change in Delphi for use as a paradigm for future process-orientated research.

► We identify residual normative and informational pressures that potentially reduce the effectiveness of Delphi and suggest ways to reduce these pressures.
► We argue that greater validity of outcomes for Delphi relative to cheaper and easier alternatives occurs through “virtuous opinion change” – greater opinion change in the direction of “truth” the further you are from it.
► Conditions for virtuous opinion change are: heterogeneous expertise; motivated and involved panelists; and feedback of good reasons for judgments or choices.
► We propose a model of opinion change in Delphi that can be used as a paradigm for future research.