Calibration of metacognitive judgments: Insights from the underconfidence-with-practice effect

• Judgments-of-learning are thought to reflect subjective probability assessments.
• Inaccurate JOLs are interpreted as reflecting under- or overconfidence.
• Underconfidence-with-practice is an example of underconfidence gleaned from JOLs.
• However, we show that UWP does not generalize beyond percentage JOL scales.
• The results cast shadow on the probability interpretation of JOLs.

Research on calibration typically compares mean judgments-of-learning (JOLs) made on a 0–100% scale with mean recall to determine whether there is overconfidence (JOL > recall), underconfidence (JOL < recall), or realism (JOL = recall). This research is founded on the assumption that JOLs directly reflect underlying representations of probability, which, if true, should mean that results from studies employing per cent JOL scales will generalize to other measures of subjective probability. We report four experiments that tested this assumption in the context of the underconfidence-with-practice (UWP) effect, the finding that JOLs underestimate recall on second and subsequent study-test cycles of multi-cycle paired-associate learning tasks (e.g., Koriat, Sheffer, & Ma’ayan, 2002). In particular, after replicating the standard UWP effect with scale JOLs in Experiment 1, we tested whether the effect would generalize to a binary JOL judgment in Experiment 2 and/or a binary betting decision in Experiment 3. In neither experiment was the UWP effect observed. Finally, to ensure that memorial and metamemorial evidence was equated for each judgment type, each item was assigned both a scale JOL and a binary betting decision in Experiment 4. UWP was observed for the 0–100% scale JOL, but not the binary betting decision. Together, the results suggest that JOLs may not directly reflect subjective probability but rather reflect an initial yes/no decision about future recall followed by a confidence rating.