Do imposters threaten data quality? An examination of worker misrepresentation and downstream consequences in Amazon's Mechanical Turk workforce

Amazon's Mechanical Turk (MTurk) is fast becoming the most popular online research platform, and as such, it is crucial for researchers to recognize its advantages and shortcomings. Here, we focused on the issue of worker deception and examined the downstream consequences of demographic misrepresentation in MTurk. In two studies, we asked: “Are we testing who we think we are testing?” and “Does demographic deception ultimately have an impact on data quality?” We found that in the presence of explicit eligibility requirements, an alarmingly high proportion of our samples misrepresented themselves in order to qualify for the studies (55.8% in Study 1 and 21.8% in Study 2). We also found that the nature of the downstream consequences of demographic deception varied across studies. Specifically, the scope of the impact may rest with the relationship between the demographic variable of interest and the outcome measure. In sum, the impact of demographic deception on data quality is multi-faceted, and a fruitful avenue of future research is to identify additional motivating factors that may underlie such deception.