Selective combination of multiple neural networks for improving model prediction in nonlinear systems modelling through forward selection and backward elimination

Combining multiple neural networks appears to be a very promising approach in improving neural network generalisation since it is very difficult, if not impossible, to develop a perfect single neural network. In the building of an aggregated neural network model, a number of individual networks are developed from different data sets and/or different training algorithms. In this paper, individual networks are developed from bootstrap re-samples of the original training and testing data sets. Instead of combining all the developed networks, this paper proposes two selective combination techniques: forward selection and backward elimination. These two techniques essentially combine those individual networks that, when combined, can significantly improve model generalisation. In forward selection, individual networks are gradually added into the aggregated network until the aggregated network error on the original training and testing data sets cannot be further reduced. In backward elimination, all the individual networks are initially aggregated and some of the individual networks are then gradually eliminated until the aggregated network error on the original training and testing data sets cannot be further reduced. The proposed techniques are applied to dynamic nonlinear process modelling and classification of diabetes database. Application results demonstrate that the proposed techniques can significantly improve model generalisation and perform better than aggregating all the individual networks and the heuristic selective combination method where networks with better performance on the training and testing data are selected.