Psychological research in the internet age: The quality of web-based data

The internet is increasingly used in psychological research to solicit participants and collect data. This paper includes two studies examining the quality of data obtained via web-based methods administered either inside or outside the lab. Both studies used item recognition accuracy as a proxy for attention to questions. Study 1 examined the extent to which undergraduate participants (N = 504) read and attended to questions either inside or outside the lab. Study 2 (N = 744) replicated Study 1, added a Mechanical Turk sample, and examined attention to non-intuitive survey instructions. Results indicated that participants demonstrated good item recognition, regardless of locale or sample; however, small sex effects on accuracy were found in both studies. Specifically, women were more accurate at identifying previously seen items than men in both Study 1 and Study 2. In Study 2, Mechanical Turk participants were more likely to read instructions than undergraduate participants, regardless of whether they participated inside or outside of the lab. The findings support the use of the internet for sampling purposes as well as survey administration, and suggest that researchers use care when studies include non-intuitive instructions.