Primal explicit max margin feature selection for nonlinear support vector machines

•We solve a primal embedded feature selection problem for nonlinear SVMs.•Use smooth hinge loss and trust region algorithm for bound constrained minimization.•Propose alternating optimization approach to break problem into two smaller ones.•Propose explicit margin maximization to solve feature selection subproblem.•Show our approach improves state-of-art and other algorithms on various data.

Embedding feature selection in nonlinear support vector machines (SVMs) leads to a challenging non-convex minimization problem, which can be prone to suboptimal solutions. This paper develops an effective algorithm to directly solve the embedded feature selection primal problem. We use a trust-region method, which is better suited for non-convex optimization compared to line-search methods, and guarantees convergence to a minimizer. We devise an alternating optimization approach to tackle the problem efficiently, breaking it down into a convex subproblem, corresponding to standard SVM optimization, and a non-convex subproblem for feature selection. Importantly, we show that a straightforward alternating optimization approach can be susceptible to saddle point solutions. We propose a novel technique, which shares an explicit margin variable to overcome saddle point convergence and improve solution quality. Experiment results show our method outperforms the state-of-the-art embedded SVM feature selection method, as well as other leading filter and wrapper approaches.