A radial basis function network classifier to maximise leave-one-out mutual information

• An orthogonal forward selection algorithm is proposed for constructing radial basis function classifiers based on maximises the leave-one-out mutual information between the classifier's predicted class labels and the true class labels.
• Integrated within each OFS step, a Bayesian procedure of hyperparameter fitting is introduced to infer the l2-norm local regularisation parameter from the data.
• The results obtained demonstrate that the proposed algorithm automatically constructs very parsimonious RBF classifiers with excellent classification generalisation performance.

We develop an orthogonal forward selection (OFS) approach to construct radial basis function (RBF) network classifiers for two-class problems. Our approach integrates several concepts in probabilistic modelling, including cross validation, mutual information and Bayesian hyperparameter fitting. At each stage of the OFS procedure, one model term is selected by maximising the leave-one-out mutual information (LOOMI) between the classifier's predicted class labels and the true class labels. We derive the formula of LOOMI within the OFS framework so that the LOOMI can be evaluated efficiently for model term selection. Furthermore, a Bayesian procedure of hyperparameter fitting is also integrated into the each stage of the OFS to infer the l2-norm based local regularisation parameter from the data. Since each forward stage is effectively fitting of a one-variable model, this task is very fast. The classifier construction procedure is automatically terminated without the need of using additional stopping criterion to yield very sparse RBF classifiers with excellent classification generalisation performance, which is particular useful for the noisy data sets with highly overlapping class distribution. A number of benchmark examples are employed to demonstrate the effectiveness of our proposed approach.

Decision boundary produced by the proposed algorithm for synthetic data set, where stars and dots represent two-class data points, respectively, while circles are the selected centres. Figure optionsDownload as PowerPoint slide