Margin-based ordered aggregation for ensemble pruning

Ensemble methods have been successfully used as a classification scheme. The reduction of the complexity of this popular learning paradigm motivated the appearance of ensemble pruning algorithms. This paper presents a new ensemble pruning method which highly reduces the complexity of ensemble methods and performs better than complete bagging in terms of classification accuracy. More importantly, it is a very fast algorithm. It consists in ordering all base classifiers with respect to a new criterion which exploits an unsupervised ensemble margin. This method highlights the major influence of low margin instances on the performance of the pruning task and, more generally, the potential of low margin instances for the design of better ensembles. Comparison to both the naive approach of randomly pruning base classifiers and another ordering-based pruning algorithm is carried out in an extensive empirical analysis.

► New measure for ranking classifiers based on an unsupervised ensemble margin. ► New ordering-based ensemble pruning algorithm favouring low margin instances. ► Pruned ensemble of 33 classifiers (average) versus 43 for accuracy ordering pruning. ► Increase in accuracy per class of up to 5% compared to accuracy ordering pruning. ► Increase in overall accuracy of up to 10% compared to complete bagging.