An experimental investigation of scanner data preparation strategies for consumer choice models

Over the past two decades, marketing scientists in academia and industry have employed consumer choice models calibrated using supermarket scanner data to assess the impact of price and promotion on consumer choice, and they continue to do so today. Despite the extensive usage of scanner panel data for choice modeling, very little is known about the impact of data preparation strategies on the results of modeling efforts. In most cases, scanner panel data is pruned prior to model estimation to eliminate less significant brands, sizes, product forms, etc., as well as households with purchase histories not long enough to provide information on key consumer behavior concepts such as loyalty, variety seeking, and brand consideration. Further, product entity aggregation is usually part of data preparation also since hundreds of SKUs are available as choice alternatives in many product categories. This study conducts an extensive simulation experiment to investigate the effects of data pruning and entity aggregation strategies on estimated price and promotion sensitivities. Characteristics of the data that may moderate the effects of data preparation strategies are also manipulated. The results show that data preparation strategies can result in significant bias in estimated parameters. Based on the results, we make recommendations on how the model builder can prepare scanner panel data so as to avoid significant biases in estimated price and promotion responses.