Consensus measurement in Delphi studies: Review and implications for future quality assurance

Consensus measurement plays an important role in Delphi research. Although it is not the technique's aim, the measurement has to be considered an important component of Delphi analyses and data interpretation. During the past 60 years, the Delphi multi-round survey procedure has been widely and successfully used to aggregate expert opinions on future developments and incidents. This paper is dedicated to how consensus (and dissent) has been measured since the technique's emergence in the 1960s and which criteria have been used. The review also includes a description of its relationship with the measurement of stability over Delphi rounds, although the major focus lies on the concept of consensus. In an extensive literature review, 15 types of measure were identified and classified for measuring consensus (and/or stability) in detail. The research reveals that there are obvious deficits in the practice and rigour of consensus measurement for Delphi research: mistakes in statistical tests or their premises have even been made. This article gives a broad understanding of the consensus concept, shows strengths and weaknesses as well as premises of different types of measure and concludes with lessons learned. Its major contribution is therefore on improving the future quality of consensus-oriented Delphi studies.

► Research reveals deficits in the practice and rigour of consensus measurement for Delphi research.
► Qualitative criteria, descriptive and inferential statistics are used for consensus measurement.
► Consensus is not a suitable stopping criterion in Delphi research.
► Consensus measurement is an important component for Delphi analysis and interpretation.