Permutation tests for goodness-of-fit testing of mathematical models to experimental data

This paper presents statistical procedures for improving the goodness-of-fit testing of theoretical models to data obtained from laboratory experiments. We use an experimental study in the expectation states research tradition which has been carried out in the “standardized experimental situation” associated with the program to illustrate the application of our procedures. We briefly review the expectation states research program and the fundamentals of resampling statistics as we develop our procedures in the resampling context. The first procedure we develop is a modification of the chi-square test which has been the primary statistical tool for assessing goodness of fit in the EST research program, but has problems associated with its use. We discuss these problems and suggest a procedure to overcome them. The second procedure we present, the “Average Absolute Deviation” test, is a new test and is proposed as an alternative to the chi square test, as being simpler and more informative. The third and fourth procedures are permutation versions of Jonckheere’s test for ordered alternatives, and Kendall’s taub, a rank order correlation coefficient. The fifth procedure is a new rank order goodness-of-fit test, which we call the “Deviation from Ideal Ranking” index, which we believe may be more useful than other rank order tests for assessing goodness-of-fit of models to experimental data. The application of these procedures to the sample data is illustrated in detail. We then present another laboratory study from an experimental paradigm different from the expectation states paradigm – the “network exchange” paradigm, and describe how our procedures may be applied to this data set.

► We show that the χ2 statistic can be misleading as a test of goodness of fit.
► We show that this problem can be rectified through permutation testing.
► WE present two new permutation tests for assessing goodness-of-fit.