Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence

The purpose of this study is to: (1) develop a ranking of peer-reviewed AI journals; (2) compare the consistency of journal rankings developed with two dominant ranking techniques, expert surveys and journal impact measures; and (3) investigate the consistency of journal ranking scores assigned by different categories of expert judges. The ranking was constructed based on the survey of 873 active AI researchers who ranked the overall quality of 182 peer-reviewed AI journals. It is concluded that expert surveys and citation impact journal ranking methods cannot be used as substitutes. Instead, they should be used as complementary approaches. The key problem of the expert survey ranking technique is that in their ranking decisions, respondents are strongly influenced by their current research interests. As a result, their scores merely reflect their present research preferences rather than an objective assessment of each journal's quality. In addition, the application of the expert survey method favors journals that publish more articles per year.

► We developed a ranking of 182 peer-reviewed Artificial Intelligence journals by surveying 873 active AI researchers.
► We compared the ranking lists developed through expert surveys and journal citation measures.
► Expert surveys and citation impact journal ranking methods cannot be used as substitutes.
► In their ranking decisions, journal quality respondents are strongly influenced by their current research interests.
► The application of the expert survey method favors journals that publish more articles per year.