Scoring procedures for multiple criteria decision aiding with robust and stochastic ordinal regression

•Single- and multi-stage ranking methods adapted for exploitation of robust and stochastic results.•Extensive experimental comparison of nineteen scoring procedures.•Five measures for quantifying the ability of different procedures to reproduce the true ranking/choice of a decision maker.•Best results obtained with exploitation of rank acceptability indices and pairwise outranking indices.

We propose several scoring procedures for transforming the results of robustness analysis to a univocal recommendation. We use a preference model in form of an additive value function, and assume the Decision Maker (DM) to provide pairwise comparisons of reference alternatives. We adapt single- and multi-stage ranking methods to select the best alternative or construct a complete ranking by exploiting four types of outcomes: (1) necessary preference relation, (2) pairwise outranking indices, (3) extreme ranks, and (4) rank acceptability indices. In each case, a choice or ranking recommendation is obtained without singling out a specific value function. We compare the proposed scoring procedures in terms of their ability to suggest the same recommendation as the one obtained with the Decision Maker׳s assumed “true” value function. To quantify the results of an extensive simulation study, we use the following comparative measures (including some newly proposed ones): (i) hit ratio, (ii) normalized hit ratio, (iii) Kendall׳s τ, (iv) rank difference measure, and (v) rank agreement measure. Their analysis indicates that to identify the best “true” alternative, we should refer to the acceptability indices for the top rank(s), whereas to reproduce the complete “true” ranking it is most beneficial to focus on the expected ranks that alternatives may attain or on the balance between how much each alternative outranks and is outranked by all other alternatives.