Regularisation methods for neural network model averaging

Data based modeling applications require the complete automation of model creation of, in general, nonlinear processes. Numerous model architectures are available to approximate complex data structures, however, creating and selecting the best model provides a present challenge for application. Model ensembles provide excellent strategies if the prediction performance of the models can be assessed in some way. Information criteria approaches, trading off goodness of fit criteria against the average variance error of the model structures, are theoretically applicable, but demonstratively perform poorly in settings with sparse data sets and complex model structures, settings which are present for many industrial purposes. This paper proposes and discusses a methodology how to regularise the weight calculation building model ensembles. Thereby, the weights trade off goodness of fit criteria against empirical measures of risk, increasing the model ensemble prediction performance. The methodology is independent of the model class, applicable to all black box models, and improves prediction performance for a variety of data in several industrial application areas.