What are confidence judgments made of? Students' explanations for their confidence ratings and what that means for calibration

Although calibration has been widely studied, questions remain about how best to capture confidence ratings, how to calculate continuous variable calibration indices, and on what exactly students base their reported confidence ratings. Undergraduates in a research methods class completed a prior knowledge assessment, two sets of readings and posttest questions, and rated their confidence in their responses to each posttest item. Students also wrote open-ended responses explaining why they marked their confidence as they did. Students provided confidence ratings on a 100-mm line for one of the passages and through magnitude scaling for the other counterbalanced passage. Calibration was calculated using a rho coefficient and yielded acceptable measures of calibration. No within-subject differences were found between 100-mm line responses and magnitude scaling responses, p = .54. Open-ended responses revealed that students base their confidence ratings on prior knowledge, characteristics of the text, characteristics of the item, guessing, and combinations of these categories. Future studies including calibration should carefully consider implicit assumptions about students' sources of confidence and how those sources theoretically relate to calibration.

► No differences were found in the distribution of 100-mm and magnitude scales in individuals' confidence ratings.
► The rho coefficient was found to be effective in calculating calibration with continuous confidence scales.
► Individuals reported using both person and task characteristics to gauge their confidence.