Check-all-that-apply (CATA) responses elicited by consumers: Within-assessor reproducibility and stability of sensory product characterizations

•Contributes to methodological validation of sensory product characterizations by consumers using CATA methodology.•A new index for within-assessor reproducibility was developed, taking all terms in a CATA question into consideration.•An index for attribute stability was developed, and revealed that CATA terms are used more/less reliably by consumers.•Test–retest reliability is attribute specific, and can vary from high to low.

In light of the growing interest in the use of check-all-that-apply (CATA) methodology for sensory product characterization, this research considered methodological aspects of importance for further uptake. Quantitative indices were developed to calculate within-assessor reproducibility and attribute stability. Across several studies (n ∼ 200), it was found that consumers are able to reproduce their sensory product characterizations. Relative to the index proposed by Campo et al. [Australian Journal of Grape and Wine Research 14 (2008) 104–115], the new global within-assessor reproducibility index is proposed as more appropriate for CATA applications that does not limit the number of terms that can be ticked as applicable and when all the terms are applicable to the product category. Heterogeneity was uncovered in within-assessor reproducibility. It was also found that product characterizations elicited by consumers were stable. There was evidence of stability in attribute use being dependent on the nature of the attribute, suggesting that the list of terms included in CATA questions is of considerable importance, and should be carefully selected. The empirical work was done in two countries and across multiple product categories, adding robustness to the findings.