Method and model comparison by sum of ranking differences in cases of repeated observations (ties)

• The procedure “sum of ranking differences” (SRD) was extended considering ties as well.
• Theoretical distribution of SRD for random numbers is derived or approximated.
• Scaling, selection of benchmark for ranking and cross-validation are discussed.
• Sensory panel- and bank indicator illustrates the SRD ranking with ties.

Significant progress has been achieved since the introduction of the new similarity measure: the sum of absolute ranking differences (SRDs) [TrAC — Trends in Anal. Chem. 29 (2010) 101–109]. Empirical evidences were accumulated about scaling, selection of the reference (benchmark) vector, cross-validation and grouping of variables (features, models, methods, etc.). The theory has been developed including the repeated observations (ties):(i)The exact theoretical distribution (null distribution) for 4 < number of objects < 9 has been calculated for SRD treatment of random numbers (it's a kind of validation, a permutation test). All possible reference vectors with ties imply different distribution.(ii)For number of objects above eight (n > 8) an approximation has been developed using the Gaussian distribution fitted on the SRD distribution given by generating of three million n-dimensional random vectors.The validity and features of the SRD methodology with ties are illustrated using two case studies: evaluation of a sensory panel and ranking of financial indicators.