A ranking-based strategy to prune variable selection ensembles

Ensemble learning has attracted significant interest in the literature of variable selection due to its great potential to reduce false discovery rate and to stabilize selection results. In this paper, a novel ensemble pruning technique called PST2E (i.e., pruned ST2E) is introduced to obtain smaller but stronger variable selection ensembles. For the ensemble members generated by the ST2E algorithm [3], we sort them in descending order according to the prediction errors associated with their determined models. Subsequently, only a desired number of members ranked ahead are integrated to compose a subensemble. On the basis of the average importance measures produced by the pruned ensemble, all candidate variables are then ranked and decided to be important or not. In the context of linear and logistic regression models, the experiments conducted with both simulated and real-world data illustrate that PST2E significantly outperforms several other popular techniques in most cases when evaluating them with multiple measures. Another advantage of PST2E is that it admits easy implementation. As a result, PST2E can be deemed as an attractive alternative to tackle variable selection tasks in real applications.