First in, best dressed: The presence of order-effect bias in journal ranking surveys

The purpose of this study is to test for the presence of order-effect bias in journal ranking surveys. Data were obtained from 379 active knowledge management and intellectual capital researchers who rated 25 journals on a 7-point scale. Five different versions of the survey instrument were utilized. Consistent with the cognitive elaboration model, the satisficing theory, and the Gricean maxim of orderliness, order-effect bias was observed in journal ranking surveys. Journals that appear in the beginning of the ranking list delivered to survey respondents consistently receive higher scores than journals at the end of the list. Overall, the position of the journal in the list explains over 10% of its score. Therefore, authors of journal ranking studies are recommended to use multiple versions of the survey instrument with randomized journal orders.

► We tested for the presence of order-effect bias in journal ranking surveys.
► We obtained data from 379 active researchers who rated 25 peer-reviewed journals.
► Order-effect bias was observed in journal ranking surveys.
► Journals that appear in the beginning of the ranking list consistently receive higher scores than journals at the end of the list.
► The position of the journal in the list explains over 10% of its score.