Measure for measure: Calibrating ten commonly used calibration scores

This study examined the dimensionality of 10 different calibration measures using confirmatory factor analysis (CFA). The 10 measures were representative of five interpretative families of measures used to assess monitoring accuracy based on a 2 (performance) × 2 (monitoring judgment) contingency table. We computed scores for each of the measures using a common data set and compared one-, two-, and five-factor CFA solutions. We predicted that the two-factor solution corresponding to measures of specificity and sensitivity used to assess diagnostic efficiency would provide the best solution. This hypothesis was confirmed, yielding two orthogonal factors that explained close to 100% of sample variance. The remaining eight measures were intercorrelated significantly with the sensitivity and specificity factors, which explained between 91 and 99 percent of variance in each measure. The two-factor solution was consistent with two different explanations, including the possibility that metacognitive monitoring may utilize two different types of processes that rely on separate judgments of correct and incorrect performance, or may be sufficiently complex that a single measurement statistic fails to capture all of the variance in the monitoring process. Our findings indicated that no single measure explains all the variance in monitoring judgments. Implications for future research are discussed.

► Compared 10 different statistical measures of calibration accuracy.
► Use innovative Monte Carlo sampling techniques to construct a very large data set.
► Confirmatory factor analyses yielded two factors that explained 100% of variance.
► Results suggest that sensitivity and specificity measures provide best fit.
► Results suggest that monitoring may utilize two independent cognitive process.