A multi-level approach using genetic algorithms in an ensemble of Least Squares Support Vector Machines

• GA is employed to guide the ensemble design in all aspects.
• The entire structure of the ensemble is optimized simultaneously.
• Multiple ways to generate diversity are explored.
• Results show the effectiveness of feature selection and ensemble pruning.

Despite the ensemble systems have been shown to be an efficient method to increase the accuracy and stability of learning algorithms in recent decades, its construction has a question to be elucidated: diversity. The disagreement among the models that compose the ensemble can be generated when they are built under different circumstances, such as training dataset, parameter setting and selection of learning algorithms. The ensemble may be viewed as a structure with three levels: input space, the base components and the combining block of the components responses. We propose a multi-level approach using genetic algorithms to build the ensemble of Least Squares Support Vector Machines (LS-SVM), performing a feature selection in the input space, the parameterization and the choice of which models will compose the ensemble at the component level and finding a weight vector which best represents the importance of each classifier in the final response of the ensemble. The combination of feature selection and parameterization should help create even more diversity. In order to evaluate the performance of the proposed approach, we use some benchmarks to compare with other classification algorithms, including some change in the fitness function of our approach.