Ranking and selection for multiple performance measures using incomplete preference information

•Ranking and selection with multiple performance measures evaluated through stochastic simulation.•Two new procedures using a multi-attribute utility function with incomplete preference information on weights related to performance measures.•Selection of pairwise and absolutely non-dominated designs based on optimal computing budget allocation.•Elicitation of weights is facilitated and computational advantages are gained compared with existing procedures.

This paper presents two new procedures for ranking and selection (R&S) problems where the best system designs are selected from a set of competing ones based on multiple performance measures evaluated through stochastic simulation. In the procedures, the performance measures are aggregated with a multi-attribute utility function, and incomplete preference information regarding the weights that reflect the relative importance of the measures is taken into account. A set of feasible weights is determined according to preference statements that are linear constraints on the weights given by a decision-maker. Non-dominated designs are selected using two dominance relations referred to as pairwise and absolute dominance based on estimates for the expected utilities of the designs over the feasible weights. The procedures allocate a limited number of simulation replications among the designs such that the probabilities of correctly selecting the pairwise and absolutely non-dominated designs are maximized. The new procedures offer ease of eliciting the weights compared with existing R&S procedures that aggregate the performance measures using unique weights. Moreover, computational advantages are provided over existing procedures that identify non-dominated designs based on the expected values of the performance measures. The new procedures allow to obtain a smaller number of non-dominated designs. They also identify these designs correctly with a higher probability or require a smaller number of replications for correct selection. Finally, the new procedures allocate a larger number of replications to the non-dominated designs that are therefore evaluated with greater accuracy. These computational advantages are illustrated through several numerical experiments.