Evaluating the model fit of diffusion models with the root mean square error of approximation

The statistical evaluation of model fit is one of the greatest challenges in the application of diffusion modeling in research on individual differences. Relative model fit indices such as the AIC and BIC are often used for model comparison, but they provide no information about absolute model fit. Statistical and graphical tests can be used to identify individuals whose data cannot be accounted for by the diffusion model, but they become overly sensitive when trial numbers are large, and are subjective and time-consuming. We propose that the evaluation of model fit may be supplemented with the root mean square error of approximation (RMSEA; Steiger & Lind, 1980), which is one of the most popular goodness-of-fit indices in structural equation modeling. It is largely invariant to trial numbers, and allows identifying cases with poor model fit, calculating confidence intervals, and conducting power analyses. In two simulation studies, we evaluated whether the RMSEA correctly rejects badly-fitting models irrespective of trial numbers. Moreover, we evaluated how variation in the number of trials, the degree of measurement noise, the presence of contaminant outliers, and the number of estimated parameters affect RMSEA values. The RMSEA correctly distinguished between well- and badly-fitting models unless trial numbers were very small. Moreover, RMSEA values were in a value range expected from structural equation modeling. Finally, we computed cut-off values as heuristics for model acceptance or rejection. In a third simulation study we assessed how the RMSEA performs in model selection in comparison to the AIC and BIC. The RMSEA correctly identified the generating model in the majority of cases, but was outperformed by the AIC and BIC. All in all, we showed that the RMSEA can be of great value in the evaluation of absolute model fit, but that it should only be used in addition to other fit indices in model selection scenarios.