Coding choices affect the analyses of a false belief measure

• In a review of over 80 papers, we demonstrate there is not agreement regarding how the unexpected contents measure is scored.
• The most commonly used scoring system for unexpected contents ignore controls for children’s memory.
• In a logit analysis of 40 papers and a data analysis of ∼1200 participants, choice of scoring system can affect analyses using this measure.
• Effect of scoring system is more pronounced when experiments have smaller sample sizes and treat age categorically.
• We recommend a set of best practices for using this measure in research, which we believe generalizes to other theory of mind measures.

The unexpected contents task is a ubiquitous measure of false belief. Not only has this measure been used to study children’s developing knowledge of belief, it has impacted the study of atypical development, education, and many other facets of cognitive development. Based on a review of articles using this task, we show that there is no consensus regarding how to score this measure. Further, examining both a logit analysis of performance on this measure and performance of a large sample of preschoolers, we show that which coding scheme researchers used to analyze raw data from this measure has a reliable effect on results, particularly when smaller sample sizes are used. Integrating our results, we conclude that the most frequently used coding scheme is flawed. We recommend best practices for scoring the unexpected contents task, and that researchers examine how they analyze data from this measure to ensure the robustness of their effects.