ExpertLens: A system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge

The complexity of policy decision-making raises the need to elicit opinions from large and heterogeneous groups of stakeholders with broad and diverse sets of expertise. Existing options for elicitation include small face-to-face panels of experts by using the Nominal Group Technique (NGT), large Delphi panels whose members do not interact with each other face-to-face, and crowdsourcing, which involves an open call for input issued to a large community of people. In an attempt to close the gap between the practical needs of policy makers and the methodological challenges associated with eliciting opinions of large, diverse, and distributed groups, we have developed a new online elicitation system and methodology called ExpertLens. By optimizing the direct interactions of NGT with the larger number of Delphi participants and the wisdom of “selected crowds,” our approach is designed to save on the costs associated with traditional expert panels, while increasing accuracy in elicitation by reducing the potential for group process losses that can occur in large, diverse, and non-collocated panels whose members interact via asynchronous online discussion boards. The ExpertLens approach is iterative, does not require participants to develop consensus, and determines what the group “thinks” by statistically analyzing data collected in all rounds of the elicitation. This paper describes the ExpertLens system and methodology, briefly discusses recent ExpertLens trials, provides conceptual arguments for why it is an appropriate model for eliciting expert opinions, illustrates its main components and analytics by using an infrastructure investment example, and discusses a research agenda for testing the underlying tenets of the ExpertLens approach.

Research highlights
► ExpertLens is a flexible online system and methodology of expert elicitation.
► It is a D-GDSS that uses modified crowdsourcing principles.
► ExpertLens uses a Delphi-like iterative approach to elicitation and feedback.
► It provides asynchronous online discussion forum for collective learning.
► It was used in studies such issues as terrorism litigation and healthcare quality improvement.