Variational Relevant Sample-Feature Machine: A fully Bayesian approach for embedded feature selection

This paper presents a Bayesian learning approach for embedded feature selection. This approach employs a fully Bayesian framework to achieve a model which is sparse in both sample and feature domains. We introduce a novel multi-step algorithm based on Variational Approximation to efficiently compute all model parameters in order to optimize the maximum a posteriori probability (MAP) measure. Experiments on both synthetic and real datasets verify that the proposed method is successful in feature selection while achieving high accuracy in both regression and classification tasks. Compared to the existing methods, especially its non-fully Bayesian counterpart, the proposed algorithm results in much higher accuracies when the size of learning data is small. Moreover, the proposed method is more reliable (evident by less variance in accuracy) than other competing algorithms.