Method and model comparison by sum of ranking differences in cases of repeated observations (ties)

•The procedure “sum of ranking differences” (SRD) was extended considering ties as well.•Theoretical distribution of SRD for random numbers is derived or approximated.•Scaling, selection of benchmark for ranking and cross-validation are discussed.•Sensory panel- and bank indicator illustrates the SRD ranking with ties.

Significant progress has been achieved since the introduction of the new similarity measure: the sum of absolute ranking differences (SRDs) [TrAC — Trends in Anal. Chem. 29 (2010) 101–109]. Empirical evidences were accumulated about scaling, selection of the reference (benchmark) vector, cross-validation and grouping of variables (features, models, methods, etc.). The theory has been developed including the repeated observations (ties):(i)The exact theoretical distribution (null distribution) for 4 < number of objects < 9 has been calculated for SRD treatment of random numbers (it's a kind of validation, a permutation test). All possible reference vectors with ties imply different distribution.(ii)For number of objects above eight (n > 8) an approximation has been developed using the Gaussian distribution fitted on the SRD distribution given by generating of three million n-dimensional random vectors.The validity and features of the SRD methodology with ties are illustrated using two case studies: evaluation of a sensory panel and ranking of financial indicators.