Learning from (failed) replications: Cognitive load manipulations and charitable giving

• Failed replications can show that seemingly arbitrary design features may impact results.
• We present a set of failed replications of a study analyzing a cognitive load manipulation and its effect on charitable giving.
• We hypothesize and test what could generate the difference in results.
• The study shows the importance of replication in advancing our understanding of the mechanisms driving a particular result.
• The study also questions the robustness of results established by cognitive load tests.

Replication of empirical studies is much more than a tool to police the field. Failed replications force us to recognize that seemingly arbitrary design features may impact results in important ways. We describe a study that used a cognitive load manipulation to investigate the role of the deliberative system in charitable giving and a set of failed replications of that study. While the original study showed large and statistically significant results, we failed to replicate using the same protocol and the same subject pool. After the first failed replication, we hypothesized that the order our study was taken in a set of unrelated studies in a laboratory session generated the differences in effects. Three more replication attempts supported this hypothesis. The study demonstrates the importance of replication in advancing our understanding of the mechanisms driving a particular result and it questions the robustness of results established by cognitive load tests.