Early stopping aggregation in selective variable selection ensembles for high-dimensional linear regression models

Nowadays, variable selection has become the most popular and effective tool to analyze high-dimensional data. Among the existing approaches, variable selection ensembles (VSEs) have exhibited their great power in improving selection accuracy and stabilizing the results of a traditional selection method. The construction of a VSE generally consists of two phases, i.e., ensemble generation and ensemble aggregation. We study selective VSEs in this paper by inserting a pruning step before combining the generated members into a VSE. As a result, a smaller but more accurate subensemble can be obtained. By taking ST2E (stochastic stepwise ensemble) as our main example, we first extended it to handle high-dimensional data. On the basis of its individuals, the aggregation order is rearranged according to their corresponding RICc (corrected risk inflation criterion) values. Then, only some members ranked ahead are averaged to estimate the importance measures for each candidate variable. In terms of several variable ranking and selection metrics, experiments conducted with simulated and real-world high-dimensional data show that pruned ST2E is superior to several other benchmark methods in most cases. By analyzing the accuracy-diversity patterns of VSEs, the pruning step is found to exclude less accurate members and lead the reserved members to more concentrate on the true importance vector.