Repeated double cross-validation for choosing a single solution in evolutionary multi-objective fuzzy classifier design

The main advantage of multi-objective genetic fuzzy systems (MoGFS) is that a number of non-dominated fuzzy rule-based systems are obtained along the tradeoff surface among conflicting objectives. Accuracy maximization, complexity minimization and interpretability maximization have often been used for multi-objective design of fuzzy rule-based classifiers. A number of non-dominated fuzzy rule-based classifiers are obtained by a single run of MoGFS. A human decision maker is supposed to choose a single final classifier from a number of obtained classifiers according to his/her preference. One problem, which has not been discussed in many studies on MoGFS, is how to choose a single final classifier. In this paper, we discuss classifier selection with no intervention of the decision maker. Whereas complexity and interpretability are very important factors in classifier selection, we concentrate on the maximization of generalization ability as the first step towards a more general handling of classifier selection. We propose the use of repeated double cross-validation (rdCV) to choose a single final classifier and to evaluate the generalization ability of the selected classifier. We also discuss how our approach can be applied to parameter specification, formulation selection and algorithm choice.