Understanding metacognitive confidence: Insights from judgment-of-learning justifications

- The delayed JOL paradigm was adapted to include written justifications of JOLs.
- Data was collected for confidence (0-100%) and binary (yes/no) JOLs.
- Justifications were analysed using a range of natural language processing techniques.
- Justifications primarily referenced cue familiarity and target accessibility.
- Justifications of confidence and binary JOLs did not directly map onto each other.

This study employed the delayed judgment-of-learning (JOL) paradigm to investigate the content of metacognitive judgments; after studying cue-target word-pairs, participants predicted their ability to remember targets on a future memory test (cued recognition in Experiments 1 and 2 and cued recall in Experiment 3). In Experiment 1 and the confidence JOL group of Experiment 3, participants used a commonly employed 6-point numeric confidence JOL scale (0-20-40-60-80-100%). In Experiment 2 and the binary JOL group of Experiment 3 participants first made a binary yes/no JOL prediction followed by a 3-point verbal confidence judgment (sure-maybe-guess). In all experiments, on a subset of trials, participants gave a written justification of why they gave that specific JOL response. We used natural language processing techniques (latent semantic analysis and word frequency [n-gram] analysis) to characterize the content of the written justifications and to capture what types of evidence evaluation uniquely separate one JOL response type from others. We also used a machine learning classification algorithm (support vector machine [SVM]) to quantify the extent to which any two JOL responses differed from each other. We found that: (i) participants can justify and explain their JOLs; (ii) these justifications reference cue familiarity and target accessibility and so are particularly consistent with the two-stage metacognitive model; and (iii) JOL confidence judgements do not correspond to yes/no responses in the manner typically assumed within the literature (i.e. 0-40% interpreted as no predictions).