Multivariate significance testing and model calibration under uncertainty

The importance of modeling and simulation in the scientific community has drawn interest towards methods for assessing the accuracy and uncertainty associated with such models. This paper addresses the validation and calibration of computer simulations using the thermal challenge problem developed at Sandia National Laboratories for illustration. The objectives of the challenge problem are to use hypothetical experimental data to validate a given model, and then to use the model to make predictions in an untested domain. With regards to assessing the accuracy of the given model (validation), we illustrate the use of Hotelling’s T2 statistic for multivariate significance testing, with emphasis on the formulation and interpretation of such an analysis for validation assessment. In order to use the model for prediction, we next employ the Bayesian calibration method introduced by Kennedy and O’Hagan. Towards this end, we discuss how inherent variability can be reconciled with “lack-of-knowledge” and other uncertainties, and we illustrate a procedure that allows probability distribution characterization uncertainty to be included in the overall uncertainty analysis of the Bayesian calibration process.