Evaluating paper and author ranking algorithms using impact and contribution awards

• Paper and author ranking algorithms are compared and evaluated.
• Large test data sets that are based on expert opinions are used.
• Using citation counts is, in general, the best ranking metric to measure high-impact papers.
• Author-level Eigenfactor performs best in ranking high-impact authors.
• Algorithms based on PageRank rank scientifically important papers better.

In the work presented in this paper, we analyse ranking algorithms that can be applied to bibliographic citation networks and rank academic entities such as papers and authors. We evaluate how well these algorithms identify important and high-impact entities.The ranking algorithms are computed on the Microsoft Academic Search (MAS) and the ACM digital library citation databases. The MAS database contains 40 million papers and over 260 million citations that span across multiple academic disciplines, while the ACM database contains 1.8 million papers from the computing literature and over 7 million citations.We evaluate the ranking algorithms by using a test data set of papers and authors that won renowned prizes at numerous computer science conferences. The results show that using citation counts is, in general, the best ranking metric to measure high-impact. However, for certain tasks, such as ranking important papers or identifying high-impact authors, algorithms based on PageRank perform better.