Aggregation of comparisons data and reversal phenomena of metrological interest

• Statistical methodology to avoid pitfalls in data amalgamation is summarised.
• Confounding interactions, spurious relationships, lurking variables are pointed-out.
• Dichotomic datasets are analysed by use of conditional probabilities.
• Design of experiment avoiding reversal phenomena of metrological import is focused.

Aggregation of comparisons data to rank experimental results and take decisions is being more and more practiced in diverse areas, spanning over a variety of disciplines including, e.g., quality function deployment in industrial engineering, scientometrics, and recovery rate testing of new medications. Problems in decision making may be accrued from the presence of hidden confounding interactions, spurious relationships, lurking variables at work. An analysis of partitioned datasets is carried-on using contingency tables and conditional probabilities. The focus is on intermediate interpretation of evidence to avoid paradoxical reversal of statistical inference when passing from sub-level data to the global level: to this aim, care in partitioning criteria is needed to balance distribution of partitioned data over successive levels, not to incur statistical dependence. An example of counter-intuitive amalgamation effects – also known as Yule-Simpson’s “paradox” – is presented and discussed, showing how to prevent such effects by proper design of experiments.