Validation of regression metamodels in simulation: Bootstrap approach

Simulation experiments are often analyzed through a linear regression model of their input/output data. Such an analysis yields a metamodel or response surface for the underlying simulation model. This metamodel can be validated through various statistics; this article studies (1) the coefficient of determination (R-square) for generalized least squares, and (2) a lack-of-fit F-statistic originally formulated by Rao [Biometrika 46 (1959) 49], who assumed multivariate normality. To derive the distributions of these two validation statistics, this paper shows how to apply bootstrapping—without assuming normality. To illustrate the performance of these bootstrapped validation statistics, the paper uses Monte Carlo experiments with simple models. For these models (i) R-square is a conservative statistic (rejecting a valid metamodel relatively rarely), so its power is low; (ii) Rao’s original statistic may reject a valid metamodel too often; (iii) bootstrapping Rao’s statistic gives only slightly conservative results, so its power is relatively high.