Assessing sensory panel performance using generalizability theory

•Algorithm for assessing the performance of individual panellists and deciding which qualify for removal from the analysis.•Reliability coefficient Φ giving important additional information about panel performance.•Statistical power analysis in combination with a reliability analysis.

Generalizability theory provides a framework for assessing panel reliability, both for the panel as a whole and for individual panellists. The variability of the sensory panel scores is split up into products, panellists, replications, and interactions between these terms. Reliability is defined as product variance over total variance. Coefficient G only includes terms including product in the denominator and focusses on the ordering of the product scores.Coefficient Φ is introduced, which includes all variance components in the denominator and focuses on the ordering as well as on the absolute values. It is shown that this latter feature provides important additional information about panel performance.An algorithm is described which excludes panellists one by one and then evaluates the contributions of each panellist on total test reliability. The focus is on changes in the coefficients and variance components after removal compared to before, allowing for an in-depth evaluation of the performance of individual panellists. When coefficient Φ increases with an amount deemed relevant (0.05 on average for all attributes, or 0.10 for a single attribute) after removal of a panellist, the panellist qualifies for exclusion from the statistical analysis. The total number of panellists to be excluded is limited to a maximum of 20% of the panel size.It is shown that a statistical power calculation is a useful addition to a reliability analysis by checking if panel discrimination meets a pre-set standard.It is explained how the reliability algorithm and the power calculation can be implemented using MS Excel.The common criteria for assessing panel performance: discrimination, consensus, and repeatability, are defined in terms of generalizability theory variance components. Discrimination focuses on maximising product variance, consensus on minimising all components containing panellist, and repeatability on minimising all components containing replicate.It is discussed how reliability results obtained using this methodology can be used for panel management.Two examples from studies carried out by the Givaudan sensory panel in Ashford, UK, are given.