A novel feature selection approach: Combining feature wrappers and filters

Feature selection is one of the most important issues in the research fields such as system modelling, data mining and pattern recognition. In this study, a new feature selection algorithm that combines feature wrapper and feature filter approaches is proposed in order to identify the significant input variables in systems with continuous domains. The proposed method utilizes functional dependency concept, correlation coefficients and K-nearest neighbourhood (KNN) method to implement the feature filter and feature wrappers. Four feature selection methods independently select the significant input variables and the input variable combination, which yields best result with respect to their corresponding evaluation function, is selected as the winner. This is similar to the basic information fusion notion of integrating the information collected from different sources. All of the four feature selection methods are performed in two stages: (i) pre-selection, (ii) selection. Two of the four feature selection methods utilize KNN method for evaluating the candidates. These two methods use sequential forward and sequential backward search mechanism, respectively, in pre-selection stage. Whereas, the third feature selection method uses correlation coefficients in the pre-selection stage. It is common to have outliers and noise in real-life data. In order to make the proposed feature selection algorithm noise and outlier resistant, approximate functional dependencies are used by utilizing membership values that inherently cope with uncertainty in the data. Thus, the fourth feature selection method makes use of approximate functional dependencies to evaluate candidates in pre-selection stage. All of these four methods apply KNN method with exhaustive search strategy in order to find the most suitable input variable combination with respect to a performance measure.