A reliability analysis of Mechanical Turk data

Amazon's Mechanical Turk (MTurk) provides researchers with access to a diverse set of people who can serve as research participants, making the process of data collection a streamlined and cost-effective one. While a small number of studies are often cited to support the use of this methodology, there remains a need for additional analyses of the quality of the research data. In the present study, MTurk-based responses for a personality scale were found to be significantly less reliable than scores previously reported for a community sample. While score reliability was not affected by the length of the survey or the payment rates, the presence of an item asking respondents to affirm that they were attentive and honest was associated with more reliable responses. Best practices for MTurk-based research and continuing research needs are addressed.