An EnKF-based scheme to optimize hyper-parameters and features for SVM classifier

The quality of models built by machine learning algorithms mostly depends on the careful tuning of hyper-parameters and feature weights. This paper introduces a novel scheme to optimize hyper-parameters and features by using the Ensemble Kalman Filter (EnKF), which is an iterative optimization algorithm used for high-dimensional nonlinear systems. We build a framework for applying the EnKF method on parameter optimization problems. We propose ensemble evolution to converge to the global optimum. We also optimize the EnKF calculation for large datasets by using the computationally efficient UR decomposition. To demonstrate the performance of our proposed design, we apply our approach for the tuning problem of Support Vector Machines. Experimental results show that the better global optima can be identified by our approach with acceptable computation cost compared to three state-of-the-art Bayesian optimization methods (SMAC, TPE and SPEARMINT).