Paired preference data with a no-preference option – Statistical tests for comparison with placebo data

• Tests for comparing 2-AC data with placebo data are discussed.
• A recently suggested test is shown to have a too high type I error rate.
• Two alternative tests are proposed with correct type I error and high power.
• One test is shown to be particularly insightful.

It is well-established that when respondents are presented with identical samples in a preference test with a no preference option, a sizable proportion of respondents will report a preference. In a recent paper (Ennis, D. M., & Ennis, J. M. (2012a). Accounting for no difference/preference responses or ties in choice experiments. Food Quality and Preference, 23, 13–17) noted that this proportion can depend on the product category, have proposed that the expected proportion of preference responses within a given category be called an identicality norm, and have argued that knowledge of such norms is valuable for more complete interpretation of 2-Alternative Choice (2-AC) data. For instance, these norms can be used to indicate consumer segmentation even with non-replicated data. In this paper, we show that the statistical test suggested by Ennis and Ennis (2012a) behaves poorly and has too high a type I error rate if the identicality norm is not estimated from a very large sample size. We then compare five χ2χ2 tests of paired preference data with a no preference option in terms of type I error and power in a series of scenarios. In particular, we identify two tests that are well behaved for sample sizes typical of recent research and have high statistical power. One of these tests has the advantage that it can be decomposed for more insightful analyses in a fashion similar to that of ANOVA F-tests. The benefits are important because they enable more informed business decisions, particularly when ingredient changes are considered for cost-reduction or health initiative purposes.