Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking-aloud protocols and eye-tracking data

Web searching for complex information requires to appropriately evaluating diverse sources of information. Information science studies identified different criteria applied by searchers to evaluate Web information. However, the explicit evaluation instructions used in these studies might have resulted in a distortion of spontaneous evaluation processes. Accordingly, the present study compared explicit evaluation instructions and neutral thinking-aloud instructions. Data from thinking-aloud protocols, eye tracking, and information problem-solving were collected from 30 participants equally distributed to two experimental conditions, that is, the Instructed Evaluation condition and the Spontaneous Evaluation condition. Instructed evaluation, as compared to spontaneous evaluation, resulted in more verbal utterances of quality-related evaluation criteria, in an increased attention focus on user ratings displayed on Web pages, and in better quality of decision making, although participants in the Instructed Evaluation condition were not able to better justify their decision as compared to participants in the Spontaneous Evaluation condition.