How to compute reliability estimates and display confidence and tolerance intervals for pattern classifiers using the Bootstrap and 3-way multidimensional scaling (DISTATIS)

When used to analyze brain imaging data, pattern classifiers typically produce results that can be interpreted as a measure of discriminability or as a distance between some experimental categories. These results can be analyzed with techniques such as multidimensional scaling (MDS), which represent the experimental categories as points on a map. While such a map reveals the configuration of the categories, it does not provide a reliability estimate of the position of the experimental categories, and therefore cannot be used for inferential purposes. In this paper, we present a procedure that provides reliability estimates for pattern classifiers. This procedure combines bootstrap estimation (to estimate the variability of the experimental conditions) and a new 3-way extension of MDS, called DISTATIS, that can be used to integrate the distance matrices generated by the bootstrap procedure and to represent the results as MDS-like maps. Reliability estimates are expressed as (1) tolerance intervals which reflect the accuracy of the assignment of scans to experimental categories and as (2) confidence intervals which generalize standard hypothesis testing. When more than two categories are involved in the application of a pattern classifier, the use of confidence intervals for null hypothesis testing inflates Type I error. We address this problem with a Bonferonni-like correction. Our methodology is illustrated with the results of a pattern classifier described by O'Toole et al. (O'Toole, A., Jiang, F., Abdi, H., Haxby, J., 2005. Partially distributed representations of objects and faces in ventral temporal cortex. J. Cogn. Neurosci. 17, 580–590) who re-analyzed data originally collected by Haxby et al. (Haxby, J., Gobbini, M., Furey, M., Ishai, A., Schouten, J., Pietrini, P., 2001. Distributed and overlapping representation of faces and objects in ventral temporal cortex. Science 293, 2425–2430).