Further investigations into the reproducibility of check-all-that-apply (CATA) questions for sensory product characterization elicited by consumers

• Reproducibility of CATA questions with consumers was assessed using a test–retest paradigm.
• Evidence of the reproducibility of CATA questions was obtained.
• Consumers’ individual reproducibility tended to be high.
• Reproducibility was affected by the size of the difference among samples.
• Presentation order of the terms did not largely affect the reproducibility of CATA questions.

The purpose of the present research was further investigate the reproducibility of check-all-that-apply (CATA) questions for sensory product characterization. Evaluations obtained when such questions are used by consumers are rarely replicated and therefore reproducibility of the data may be at risk. Results from the present work, which included five studies, each with 100–200 consumers across a range of product categories, revealed that sensory product characterizations obtained using CATA questions with consumers are highly reproducible. Hence, the research confirms previous research by Jaeger, Chheang, et al. (2013) and extends it to the use of CATA terms in randomised presentation order as has been recommended to avoid satisficing response behavior. In the future, if CATA studies are conducted without replication and researches seek to examine the reliability of CATA data, the use of a posterior bootstrapping re-sampling approach is suggested.