Easy and powerful analysis of replicated paired preference tests using the χ2 test

Assessor heterogeneity in replicated paired preference testing might mislead to the conclusion that there are no product differences at all, in particular if two equally sized consumer segments with opposed preferences occur. Different parametric approaches to deal with heterogeneity have been proposed, one of which is fitting a Beta-binomial model. Alternatively, we propose to use the ordinary χ2-goodness-of-fit test to globally test for product differences. Examples are used to illustrate how the intermediate results of this test offer additional insight into the data and allow identifying possible consumer segments. Simulations show that the χ2 test is more powerful than both the ordinary binomial and the Beta-binomial test, even if the data are truly Beta-binomially distributed. As the χ2 approximation is liberal in many settings, we recommend using the Monte-Carlo simulation-based version of this test. We can easily perform this by using the open source software R.