Heterogeneity in the consistency of best–worst scale responses

•Most of the inconsistency between best and worst responses in BW experiments can be attributed to a sub-set of the sample.•Approximately 25% of a research sample accounts for 50–60% of the inconsistencies between best and worst responses.•There is no clear reason why people are inconsistent. They are attentive and take the same amount of time to complete the experiment.•Analysis must accommodate heterogeneity in BW consistency for accurate prediction using BW data.

Best–worst Scaling is one of the dominant measurement approaches in choice experimentation. When employed it provides substantial information on peoples' preferences without making choice tasks prohibitively long. However, one concern with this method is that peoples' selection of a best may not reflect the same preferences as when a worst is selected. Research into such an inconsistency between best and worst responses has found it to be a non-trivial and persistent problem. This research further investigates these inconsistencies and finds that they can largely be attributed to a relatively small group of people in the sample who do not anchor their worst responses onto their best responses as literature suggests they would. In fact, 25% of the participants in a sample account for between 50 and 60% of the inconsistent responses recorded. The presence of this group and their disproportionate contribution to the number of inconsistencies in best and worst responses provide strong evidence that there is heterogeneity in how consistently people formulate responses in best–worst tasks. Recommendations are made regarding how to accommodate this phenomenon in utility based choice models so that better predictions of choices can be made.

Chinese abstract在选择实验中，最佳、最差级别是占主导地位的度量方法之一。使用中，它提供了有关人们喜好的实质性信息，且不会使选择任务过于漫长。然而，关于此方法的一种担忧是人们所选择的最佳方式，可能并不反映选择最差方式时的相同的偏好。对此种最佳、最差的回答之间的不一致性的研究，已经发现这是个不小的、持久性的问题。该研究进一步调查了这种不一致性，并发现它们在很大程度上可能归因于样品中的相对较小的一群人，他们并不像文献表述的会将最差的回答固定到最佳的回答上。事实上，在一个样品帐户中，25％的参与者占了所记录的50至60％的不一致回答。这个群体的存在，以及他们对最佳和最差的回答的不一致的数量的不成比例的影响，提供了强有力的证据，证明在人们如何始终如一地规定最佳、最差任务中的回答时，存在异质性。本文就如何在基于效用的选择模型中适应这一现象给出了建议，以更好的预测选择。