Should we trust survey data? Assessing response simplification and data fabrication

• Important international surveys can generate dynamics that undermine data integrity.
• Even among well-educated respondents extreme response simplification occurs often.
• We uncovered strong evidence of data fabrication in several countries in PISA 2009.
• Unit nonresponse rates are, at best, ambiguous indicators of data quality.

While many factors, such as unit- and item nonresponse, threaten data quality, we focus on data contamination that arises primarily from task simplification processes. We argue that such processes can occur at two levels. First, respondents themselves may engage in various response strategies that minimize their time and effort in completing the survey. Second, interviewers and other employees of the research institute might take various shortcuts to reduce their time and/or to fulfill the requirements of their contracts; in the simplest form this can be done via copy-and-paste procedures.This paper examines the cross-national quality of the reports from principals of schools participating in the 2009 PISA. We introduce two measures of data quality to document that extreme response simplification characterizes the behavior of substantial numbers of school principals in numerous countries. Additionally, we discovered strong evidence of data fabrication in several countries.